Automatic actions based on contextual replies

ABSTRACT

A computing device includes at least one processor and at least one module, operable by the at least one processor to receive an communication, determine, based at least in part on the communication, one or more candidate responses to the communication, and receive an indication of user input that selects a candidate response from the one or more candidate responses. The at least one module may be further operable by the at least one processor, responsive to receiving the indication of user input that selects the candidate response, to send the candidate response, determine, based at least in part on at least one of the candidate response and the communication, an operation that is contextually related to the candidate response, and execute the operation.

This application is a continuation of U.S. application Ser. No.14/258,692, filed Apr. 22, 2014, the entire content of which is herebyincorporated by reference.

BACKGROUND

Many computing devices enable a user to respond to receivedcommunications sent by users of other computing devices. Some suchcomputing devices provide one or more stock responses, stored by thecomputing device, that a user may choose from when composing a responseto the received communication. In response to receiving an indication ofuser input that selects a particular stock response, the computingdevice may send the selected stock response from the recipient'scomputing device to the computing device of a different user that sentthe received communication.

SUMMARY

In one example, a method includes receiving, by a computing device, acommunication, determining, based at least in part on the communication,one or more candidate responses to the communication, and receiving, bythe computing device, an indication of user input that selects acandidate response from the one or more candidate responses. The methodmay further include, responsive to receiving the indication of userinput that selects the candidate response, sending, by the computingdevice, the candidate response, determining, based at least in part onat least one of the candidate response and the communication, anoperation that is contextually related to the candidate response, andexecuting, by the computing device, the operation.

In another example, a computing device includes at least one processorand at least one module, operable by the at least one processor toreceive a communication, determine, based at least in part on thecommunication, one or more candidate responses to the communication, andreceive an indication of user input that selects a candidate responsefrom the one or more candidate responses. The at least one module may befurther operable by the at least one processor, responsive to receivingthe indication of user input that selects the candidate response, tosend the candidate response, determine, based at least in part on atleast one of the candidate response and the communication, an operationthat is contextually related to the candidate response, and execute theoperation.

In another example, a computer-readable storage medium is encoded withinstructions that, when executed, cause at least one processor toreceive a communication, determine, based at least in part on thecommunication, one or more candidate responses to the communication, andreceive an indication of user input that selects a candidate responsefrom the one or more candidate responses. The computer-readable storagemedium may be further encoded with instructions that, when executed,cause the at least one processor, responsive to receiving the indicationof user input that selects the candidate response, to send the candidateresponse, determine, based at least in part on at least one of thecandidate response and the communication, an operation that iscontextually related to the candidate response, and execute theoperation.

The details of one or more examples are set forth in the accompanyingdrawings and the description below. Other features, objects, andadvantages will be apparent from the description and drawings, and fromthe claims.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a block diagram illustrating an example computing device andgraphical user interfaces (GUIs) for performing contextually relatedoperations responsive to sending a response to a received communication,in accordance with one or more techniques of the present disclosure.

FIG. 2 is a block diagram illustrating details of one example of acomputing device for performing contextually related operationsresponsive to sending a response to a received communication, inaccordance with one or more techniques of the present disclosure.

FIG. 3 is a block diagram illustrating an example computing device thatoutputs graphical content for display at a remote device, in accordancewith one or more techniques of the present disclosure.

FIG. 4 is a block diagram illustrating an example computing device andGUIs for performing contextually related operations responsive tosending a response to a received communication, in accordance with oneor more techniques of the present disclosure.

FIG. 5 is a block diagram illustrating an example computing device andGUIs for performing contextually related operations responsive tosending a response to a received communication, in accordance with oneor more techniques of the present disclosure.

FIG. 6 is a flow diagram illustrating example operations of a computingdevice for performing contextually related operations responsive tosending a response to a received communication, in accordance with oneor more techniques of the present disclosure.

DETAILED DESCRIPTION

In general, techniques of this disclosure are directed to providingcandidate responses in reply to a received communication andautomatically performing one or more operations that are contextuallyrelated to at least one selected candidate response. That is, accordingto techniques of the present disclosure, a computing device may outputone or more candidate responses that are relevant to a receivedcommunication and, responsive to receiving a selection of a particularcandidate response, perform one or more operations or “actions” relatedto the selected candidate response. As an example, a computing devicemay not only send a particular candidate response upon selection of thatcandidate response by a user, but also create a new calendar event for ameeting associated with the received communication and/or the particularcandidate response.

In this way, computing devices implementing techniques of thisdisclosure may reduce or eliminate the need for the user to provideadditional input to cause the computing device to perform the relevantoperations. Such functionality may be advantageous when it is difficultor undesirable for a user to manually provide input to a computingdevice.

FIG. 1 is a block diagram illustrating an example computing device 2 andgraphical user interfaces (GUIs) 20, 26, 30 for performing contextuallyrelated operations responsive to sending a response to a receivedcommunication, in accordance with one or more techniques of the presentdisclosure. Examples of computing device 2 may include, but are notlimited to, portable, mobile, or other devices, such as mobile phones(including smartphones), laptop computers, desktop computers, tabletcomputers, smart television platforms, personal digital assistants(PDAs), server computers, mainframes, and the like. For instance, in theexample of FIG. 1, computing device 2 may be a wearable computingdevice, such as a smartwatch.

Computing device 2, as shown in the example of FIG. 1, includes userinterface (UI) device 4. UI device 4 of computing device 2 may beconfigured to function as an input device and/or an output device forcomputing device 2. UI device 4 may be implemented using varioustechnologies. For instance, UI device 4 may be configured to receiveinput from a user through tactile, audio, and/or video feedback.Examples of input devices include a presence-sensitive display, apresence-sensitive or touch-sensitive input device, a mouse, a keyboard,a voice responsive system, video camera, microphone or any other type ofdevice for detecting a command from a user. In some examples, apresence-sensitive display includes a touch-sensitive orpresence-sensitive input screen, such as a resistive touchscreen, asurface acoustic wave touchscreen, a capacitive touchscreen, aprojective capacitance touchscreen, a pressure sensitive screen, anacoustic pulse recognition touchscreen, or another presence-sensitivetechnology. That is, UI device 4 of computing device 2 may include apresence-sensitive device that may receive tactile input from a user ofcomputing device 2. UI device 4 may receive indications of the tactileinput by detecting one or more gestures from the user (e.g., when theuser touches or points to one or more locations of UI device 4 with afinger or a stylus pen).

UI device 4 may additionally or alternatively be configured to functionas an output device by providing output to a user using tactile, audio,or video stimuli. Examples of output devices include a sound card, avideo graphics adapter card, or any of one or more display devices, suchas a liquid crystal display (LCD), dot matrix display, light emittingdiode (LED) display, organic light-emitting diode (OLED) display, e-ink,or similar monochrome or color display capable of outputting visibleinformation to a user of computing device 2. Additional examples of anoutput device include a speaker, a cathode ray tube (CRT) monitor, aliquid crystal display (LCD), or other device that can generateintelligible output to a user. For instance, UI device 4 may presentoutput to a user of computing device 2 as a graphical user interfacethat may be associated with functionality provided by computing device2. In this way, UI device 4 may present various user interfaces ofapplications executing at or accessible by computing device 2 (e.g., anelectronic message application, an Internet browser application, etc.).A user of computing device 2 may interact with a respective userinterface of an application to cause computing device 2 to performoperations relating to a function.

In some examples, UI device 4 of computing device 2 may detecttwo-dimensional and/or three-dimensional gestures as input from a userof computing device 2. For instance, a sensor of UI device 4 may detectthe user's movement (e.g., moving a hand, an arm, a pen, a stylus, etc.)within a threshold distance of the sensor of UI device 4. UI device 4may determine a two or three dimensional vector representation of themovement and correlate the vector representation to a gesture input(e.g., a hand-wave, a pinch, a clap, a pen stroke, etc.) that hasmultiple dimensions. In other words, UI device 4 may, in some examples,detect a multi-dimension gesture without requiring the user to gestureat or near a screen or surface at which UI device 4 outputs informationfor display. Instead, UI device 4 may detect a multi-dimensional gestureperformed at or near a sensor which may or may not be located near thescreen or surface at which UI device 4 outputs information for display.

In the example of FIG. 1, computing device 2 includes user interface(UI) module 6, device location module 8, application modules 10A-10N(collectively “application modules 10”), and communication action module12. Modules 6, 8, 10, and/or 12 may perform one or more operationsdescribed herein using hardware, software, firmware, or a mixturethereof residing within and/or executing at computing device 2.Computing device 2 may execute modules 6, 8, 10, and/or 12 with oneprocessor or with multiple processors. In some examples, computingdevice 2 may execute modules 6, 8, 10, and/or 12 as a virtual machineexecuting on underlying hardware. Modules 6, 8, 10, and/or 12 mayexecute as one or more services of an operating system or computingplatform or may execute as one or more executable programs at anapplication layer of a computing platform.

UI module 6, as shown in the example of FIG. 1, may be operable bycomputing device 2 to perform one or more functions, such as receiveinput and send indications of such input to other components associatedwith computing device 2, such as modules 8, 10, and/or 12. UI module 6may also receive data from components associated with computing device2, such as modules 8, 10, and/or 12. Using the data received, UI module6 may cause other components associated with computing device 2, such asUI device 4, to provide output based on the received data. For instance,UI module 6 may receive data from one of application modules 10 todisplay a GUI.

Application modules 10, as shown in the example of FIG. 1, may includefunctionality to perform any variety of operations on computing device2. For instance, application modules 10 may include a word processor, anemail application, a web browser, a multimedia player, a calendarapplication, an operating system, a distributed computing application, agraphic design application, a video editing application, a webdevelopment application, or any other application. In some examples, oneor more of application modules 10 may be operable to receivecommunications from other devices, such as email messages, calendaralerts or meeting requests, or other communications. For instance, oneof application modules 10 (e.g., application module 10A) may be a textmessaging (e.g., SMS/MMS) application. Application module 10A mayinclude functionality to compose and send communications, receivecommunications, respond to received communications, and other functions.

In the example of FIG. 1, communication action module 12 may be operableto determine candidate responses to received communications anddetermine one or more operations that are contextually related to aresponse communication, in accordance with the techniques describedherein. That is, communication action module 12 may includefunctionality to provide one or more relevant messages that a user mayselect to cause computing device 2 to send the message as a response toa received communication. Furthermore, communication action module 12may include functionality to execute an operation that is relevant,based at least in part on the received communication and/or acommunication sent in response to the received communication, to thecontext of the correspondence. Relevance (e.g., of an operation or of aresponse message), generally, may be an indication of semanticsimilarity, contextual similarity, or any other type of similarity.Relevance, in some examples, may be represented by a probability valueor score that indicates a level of similarity between two objects, or alevel of similarity between two groups of objects (e.g., the number ofobjects that exist in both groups, the number of objects in one groupthat exist in the other group, the percentage of objects that exist inboth groups, or other value).

Communication action module 12 may determine messages a user can selectin response to a received communication and, in addition to sending aselected response, communication action module 12 may cause computingdevice 2 to perform an operation related to the sent response. Furtherdetails and example operations of communication action module 12 aredescribed with respect to the following examples.

Computing device 2 may, in the example of FIG. 1, receive acommunication. A received communication may include information (e.g.,generated based on input provided by a user of another device). Examplesof information that may be included in a received communication includetext (e.g., letters, words, numbers, punctuation, etc.), emoji, imagesor icons, video, audio, or other information. The received communicationmay be structured or formatted according to one or more protocols. Forinstance, a received communication may be an SMS message. The SMSmessage may include the textual information “We're all at Yotteru SushiRestaurant.”

In the example of FIG. 1, responsive to receiving a communication,computing device 2 may output an indication of the receivedcommunication for display. In general, an indication of a receivedcommunication may be any visual representation of the communication,such as a notification or other visual object output for display as partof a GUI. Computing device 2 may provide the message to one or more ofapplication modules 10 that are specified to handle SMS messages, suchas application module 10A. Application module 10A may receive the SMSmessage and may cause one or more other components of computing device 2to output an indication of the message (e.g., for display to a user ofcomputing device 2). That is, responsive to receiving the SMS message,application module 10A may send data to UI module 6 to cause UI device 4to display GUI 20. As shown in the example of FIG. 1, GUI 20 includes anindication of the information included in the SMS message (e.g., text22, “We're all at Yotteru Sushi Restaurant”) and a user-selectableelement 24 to reply to the message. In some examples, an applicationmodule may cause additional or other information to be output fordisplay, such as a time at which the message was received.

Computing device 2 may, in the example of FIG. 1, receive an indicationof input that instructs computing device 2 to enable a user to composeand/or select a response to the received communication. For instance, auser of computing device 2 may perform input 25 at UI device 4. UIdevice 4 may detect input 25 and send an indication of the input to UImodule 6. UI module 6 may provide data to application module 10A basedon the received indication and application module 10A may determine thatinput 25 corresponds to a selection of element 24.

In the example of FIG. 1, computing device 2 may determine one or morecandidate responses to the received communication, based at least inpart on the received communication. Responsive to receiving dataindicating a user's selection to respond to the SMS message (e.g., anindication of input 25), application module 10A may communicate withcommunication action module 12 to obtain at least one candidateresponse. A candidate response may be a message that a user may selectas a response to a received communication. For instance, in the exampleof FIG. 1, each candidate response determined by communication actionmodule 12 may be a message that computing device 2 may send, based on auser's selection, in order to respond to the SMS message, “We're all atYotteru Sushi Restaurant.”

In order to obtain candidate responses, application module 10A mayrequest candidate responses from communication action module 12. In someexamples, a request for candidate responses to a received communicationmay include at least a portion of the information included in thereceived communication. In some examples, a request may includeadditional or other information, such as communication metadata,location information, user information, other communications (e.g.,other messages in the thread or chain of communications between the userof computing device 2 and the sender of the received communication), orother information about the received communication, about the computingdevice, or about a user of the computing device. In some examples,application modules 10 may only generate a request and/or obtainpersonal data (e.g., information included in the received communicationand/or other information) if a user of computing device 2 providesexplicit permission.

Communication action module 12 may receive a request from application10A and determine candidate responses. In some examples, communicationaction module 12 may determine candidate responses based at least inpart on the subject matter of the received communication. That is,communication action module 12 may, in some examples, determine one ormore subjects included in the received communication and use thatinformation to predict candidate responses. For instance, communicationaction module 12 may perform one or more text analysis and/or naturallanguage processing techniques (e.g., clustering, noun phraseextraction, etc.) on the received communication to determine the subjector subjects of the communication (e.g., semantic elements). A subject,in general, may be a person, place, item or object, etc. Examples of asemantic element may include a particular store or restaurant, a chainof stores or chain of restaurants, a geographical location (e.g., acity, a state, a country, etc.), a person, an object (e.g., a hockeystick, a watch, a television, a dog, etc.), a concept (e.g., pregnancy,milkshakes, existentialism, etc.), any combination or type thereof, orany other focus of information. In some examples, a collection ofsemantic elements that are included in a communication may represent asemantic meaning of the communication. That is, a semantic meaning may,in some examples, be based on the subject or subjects (e.g., items,places, concepts, etc.) included in the communication.

In some examples, communication action module 12 may determine candidateresponses by using one or more lookup tables. For instance,communication action module 12 may determine whether the receivedcommunication includes one or more specific words and/or punctuationmark combinations from the lookup table, such as a question word or aquestion mark character. If the received communication includes thespecific combination of words and/or punctuation marks, communicationaction module 12 may provide the associated candidate responses from thelookup table. If the input to the lookup table is the word “are” and aquestion mark character, for instance, the lookup table may providecandidate responses such as “Yes,” “No,” etc. In other words, whiledescribed in some examples as predicting candidate responses,communication action module 12 may determine candidate responses is anynumber of ways, such as providing stored candidate responses using alookup table, predicting candidate responses based on text analysis, orother ways. Further details of such various techniques are described inFIG. 2.

In some examples, communication action module 12 may determine candidateresponses based additionally or alternatively on previously selectedresponses. That is, to determine candidate responses, communicationaction module 12 may, in some examples, use information that indicatespreviously received communications (or subjects thereof) and thecommunications that have been selected as responses to the previouslyreceived communications. By analyzing the received communications andthe corresponding selected responses, communication action module 12 maydetermine likely replies to common messages and/or messages havingcommon subjects. For instance, communication action module 12 mayreceive a request including the information, “Are you coming?”Communication action module 12 may analyze the text to determine thatthe received communication is a request for the status of the recipient.Based on the subject of the text, communication action module 12 maydetermine that the most common replies to similar messages are “I'll beright there,” “yes,” and “no.”

In some examples, the information included in a received communicationand/or the candidate responses need not be proper language. Forinstance, communication action module 12 may receive a request todetermine candidate responses that includes the text, “<3” (e.g., aheart emoticon). In such instance, communication action module 12 maydetermine that the most common responses are “<3”, and “:)” (e.g., asmiley emoticon).

In some examples, communication action module 12 may determine candidateresponses based on communications previously received by computingdevice 2 and corresponding responses previously selected by the user ofcomputing device 2. That is, in some examples communication actionmodule 12 may determine candidate responses based on communication dataassociated with the user of computing device 2. In some examples,communication action module 12 may determine candidate responses basedon previously received communications and corresponding responses thatare associated with more users. For instance, communication actionmodule 12 may cause computing device 2 to communicate with anothercomputing device (e.g., a server system) to obtain and and/or maintaininformation including communication data from multiple computingdevices. That is, in some examples computing device 2 may determinecandidate responses based on aggregate communication data associatedwith a plurality of users (e.g., the user of computing device 2 and/orother users). In such examples, information about a user's receivedcommunications and/or selected response communications may only be usedif that user explicitly provides permission (e.g., via his or herrespective computing device). Furthermore, any data may be anonymizedbefore transmittal and/or use. That is, information that could identifyor categorize a user may be stripped or generalized before providing thecommunications data (e.g., to the server system).

In some examples, candidate responses may be responses that have beenpreviously selected by a user at least at a threshold frequency. Thatis, a candidate response may, in some examples, represent a responsethat was previously provided as a candidate response a number of timesand was selected by users a sufficient number of those times to exceed athreshold frequency (e.g., at least 10%, at least 20%, etc.) ofselection. In other examples, candidate responses may be basedadditionally or instead on manual review by developers or administrators(e.g., of the server system). Manual review may be beneficial, forexample, to ensure candidate responses use appropriate language and/orare accurate.

In some examples, communication action module 12 may determine candidateresponses based additionally or alternatively on contextual informationabout computing device 2 and/or a user of computing device 2 (e.g.,information that defines a context of the computing device). Contextualinformation, in general, may include any information about the computingdevice, an environment of the computing device, and/or about a user ofthe computing device.

In some examples, items of contextual information may be collections ofdata (e.g., a text data structure, a numeric data structure, or otherdata structure) that represents a location of the computing device(e.g., a GPS location); information indicating a time as determined bythe computing device; information indicating one or more applicationsinstalled at the computing device; information indicating one or moreapplications currently executing at the computing device; informationindicating one or more networks (e.g., wireless networks) available tothe computing device; data that represents one or more other computingdevices in proximity to the computing device (e.g., within 10 feet,within 100 feet, or other distance); data that represents an operatingmode (e.g., silent mode, airplane mode, driving mode, standby mode, lowbattery mode, or any other mode of operation) of the computing device,data obtained from one or more sensors of the computing device (e.g.,temperature data, ambient noise level data, light level data,acceleration/movement data, image/video data, and other data), or anyother data about the status or current state of the computing device.

In some examples, an item of contextual information may additionally oralternatively be information about a user of the computing device, suchas a name of the user, a user identification (UID) of the user,information from one or more social media network service accountsassociated with the user, information from one or more calendars orscheduling applications associated with the user, information indicatingone or more social or professional relationships of the user (e.g., usercontacts), or any other information about the user.

In some examples, communication action module 12 may receive contextualinformation (e.g., one or more items of contextual information) with arequest for candidate responses. For instance, communication module 12may receive a request from one of application modules 10 that includesan indication of a particular communication as well as one or more itemsof contextual information each indicating a location of computing device2. Communication module 12 may provide candidate responses based on theindication of the particular communication as well as on the locations.

In some examples, communication action module 12 may additionally oralternatively obtain contextual information from other sources. Forinstance, communication module 12 may receive explicit permission from auser (e.g., the user of computing device 2) to access communicationsinformation, social media network service account information, or otherinformation pertaining to the user. That is, communication action module12 may obtain contextual information about computing device 2,information about the user of computing device 2, and/or otherinformation for use in determining candidate responses. In someexamples, communication action module 12 may determine candidateresponses based on obtained contextual information by determiningselected responses that correspond to a similar context.

In the example of FIG. 1, communication action module 12 may determineone or more candidate responses to the SMS message, “We're all atYotteru Sushi Restaurant” and send at least an indication of thecandidate responses to application module 10A. Application module 10Amay receive the indication of the determined candidate responses andsend data to UI module 6 to cause UI device 4 to display at least one ofthe candidate responses. For instance, application module 10A may senddata to cause UI device 4 to output GUI 26.

GUI 26, as shown in the example of FIG. 1, includes response options28A-28C (collectively, “response options 28”). Response options 28 mayeach represent a candidate response received from communication actionmodule 12. That is, responsive to receiving a request for candidateresponses to the SMS message “We're all at Yotteru Sushi Restaurant,”communication action module 12 may determine three candidate responses:“On my way,” “I can't make it,” and “The one on Washington St?”corresponding to response options 28A, 28B, and 28C, respectively.

While each representing a candidate response received from communicationaction module 12 in the example of FIG. 1, one or more of responseoptions 28 may, in other examples, represent other response options,such as responses provided by application module 10A. In some examples,an application module may modify one or more received candidateresponses based on various types of information, and one or more ofresponse options 28 may represent a modified candidate response. In anycase, response options 28 of GUI 26 may enable the user of computingdevice 2 to select a candidate response to cause computing device 2 tosend in response to the received SMS message. In some examples, GUI 26may include a manual entry option. A manual entry option may enable theuser of computing device 2 to input (e.g., using voice input, touchinput, or other input) a custom response. A custom response may be usedwhen, for example, the responses indicated by one of response options 28are incorrect or inaccurate.

In the example of FIG. 1, computing device 2 may receive an indicationof user input that selects a candidate response from the one or morecandidate responses. An indication of user input, generally, may be datarepresenting input provided by a user at one or more input devices, suchas touch or haptic input, voice input or other audio input, or any otherform of input. For instance, the user of computing device 2 may provideinput 29 at UI device 4, which may include a tap gesture at or near thelocation of UI device 4. UI device 4 may detect input 29 and send anindication of the input to UI module 6. UI module 6 may provide data toapplication module 10A based on the received indication, and applicationmodule 10A may determine that input 29 corresponds to a selection ofresponse option 28A.

Responsive to receiving the indication of user input that selects thecandidate response, computing device 2 may, in the example of FIG. 1,send the candidate response. That is, selecting one of response options28 or inputting a custom response may cause application module 10A torespond to the received communication by sending the selected responseto one or more other computing devices, which may include a computingdevice of the recipient for the message. For instance, responsive toreceiving an indication of input 29 that selects response option 28A,application module 10A may send the candidate response corresponding toresponse option 28A (e.g., “On my way”). In some examples, the selectedresponse may be sent directly to the computing device from which the SMSmessage was received. In some examples, the selected response may besent to additional or other computing devices, such as one or moredevices that, together, represent a network to which computing device 2is connected (e.g., a cellular or wireless network). That is, computingdevice 2 may send the selected response to the computing device fromwhich the SMS message was received by sending the response to networkdevices or other devices situated along a route between the twocomputing devices. In some examples, the selected response mayadditionally or alternatively be sent to other devices, such as a serversystem or a cloud computing environment, for subsequent use in providingimproved candidate responses.

In the example of FIG. 1, responsive to receiving the indication of userinput that selects the candidate response, computing device 2 maydetermine, based at least in part on at least one of the candidateresponse and the received communication, an operation that iscontextually related to the candidate response. In other words,computing device 2 determines the operation based at least in part onthe candidate response, based at least in part on the receivedcommunication, or based at least in part on both the candidate responseand the received communication. An operation contextually related to thecandidate response may be executable by a computing device (e.g.,computing device 2 or other computing device) to perform an action inaddition to generating, displaying, and/or sending a response to thereceived communication. That is, operations contextually related to aresponse, as defined herein, may not include one or more of: determiningcandidate responses to a received communication, outputting thecandidate responses (e.g., for display to a user), or sending a responseto a received communication. Rather, an operation that is contextualrelated to a response may represent an operation that a user mightotherwise manually cause a computing device to perform if he or shereceived the communication and sent the corresponding response. Forinstance, operations that are contextually related to a response mayinclude operations that cause a computing device to provide directionsto a location indicated in the received communication and/or theresponse, operations that cause the computing device to create, modify,or delete a calendar item corresponding to an event in the receivedcommunication or response, and operations that cause the computingdevice to provide other information about a subject or subjects (e.g., alocation, a person, an item, an event, etc.) of the receivedcommunication and/or the response. Further examples of operations thatare contextually related to a response may include operations that causea computing device to initiate a phone call, create a reservation for arestaurant, a rental car, an airline flight, or hotel, operations thatcause the computing device to execute a search for items on one or morenetworks (e.g., the Internet) or websites (e.g., auction websites,social network service websites, etc.).

Application module 10A may receive an indication of input 29 to selectresponse option 28A and may send an indication of the correspondingcandidate response to communication action module 12. Responsive toreceiving an indication of a selected response, communication actionmodule 12 may, in some examples, determine one or more subjects of theselected response. For instance, communication action module 12 mayperform text analysis and/or natural language processing techniques onthe selected response to determine the subject or subjects of thecommunication and/or determine other factors. That is, in some examples,communication action module 12 may determine the subject matter of boththe received communication and the response to the receivedcommunication in order to determine an operation contextually related tothe response.

In some examples, communication action module 12 may additionally oralternatively determine and/or update the context of computing device 2in response to receiving the indication of the selected response. Forinstance, communication action module 12 may determine whether anycontextual information about computing device 2 and/or the user ofcomputing device 2 (e.g., the location of computing device 2, the socialnetwork service account of the user, etc.) has changed and, if anyinformation has changed, communication action module 12 may update thecontextual information accordingly.

In some examples, communication action module 12 may determine anoperation that is contextually related to a selected candidate responseby predicting one or more operations that a user would likely cause acomputing device to perform, given one or more of the determined subjector subjects of the received communication, the determined subject orsubjects of the selected candidate response, and/or the obtainedcontextual information. For instance, communication action module 12 maydetermine operations that were previously performed by computing device2 and/or other computing devices in contexts that were similar to apresently determined context of the computing device 2. That is,communication action module 12 may determine contextually relatedoperations based on previous operations performed by computing devicesin a similar context after receiving a similar communication andreceiving input from a user to cause the computing device to generateand/or send a similar response.

In other examples, communication action module 12 may determine anoperation that is contextually related to a selected candidate responsein various other ways. For instance, communication action module 12 mayemploy a lookup table that provides associations between information inthe communications and contextually related operations. That is,communication action module 12 may search the received communicationand/or the selected candidate response for one or more words and/or oneor more punctuation marks. If the communications include words,punctuation marks, or a combination thereof that exist in the lookuptable, communication action module 12 may determine the correspondingoperation. As one example, if the received communication and/or theselected candidate response include the word “weather,” communicationaction module 12 may determine an operation to output weatherinformation for display (e.g., at UI device 4). In other words, whiledescribed in some examples as predicting operations that arecontextually related to a selected candidate response, communicationaction module 12 may determine contextually related operations in anynumber of ways, such as obtaining stored operations using a lookuptable, predicting contextually related operations based on textanalysis, or other ways.

Communication action module 12 may, in the example of FIG. 1, receivethe indication of the selected response, “On my way,” and determine anoperation that is related to the selected response. For instance,communication action module 12 may predict an operation that the user ofcomputing device 2 may cause computing device 2 to perform responsive tothe user receiving the SMS message “We're all at Yotteru SushiRestaurant” and selecting the response “On my way.” That is,communication action module 12 may determine a subject or subjects ofthe selected response and, based on the subject(s) of the selectedresponse, the subject(s) of the received communication and/or thecontextual information, communication action module 12 may determinethat the user of computing device 2 is likely going to travel to alocation indicated in the received communication. Furthermore,communication action module 12 may determine that it is unlikely thatthe user has travelled to the location recently, and thus it is likelythat the user will provide input to cause computing device 2 to providedirections to the location. Therefore, communication action module 12may determine an operation to provide directions to the location.

In some examples, communication action module 12 may determine whatapplication modules are installed at computing device 2 (e.g., as partof determining the operation that is contextually related to theselected response or at some earlier point in time that is not based ondetermining the operation). That is, communication action module 12 may,in some examples, determine what application modules are available atcomputing device 2 to perform determined operations. In some examples,communication action module 12 may determine more likely operations thatcan be performed by application modules 10, and determine less likelyoperations, e.g., operations that cannot be performed by applicationmodules 10. For instance, if a maps application is not included inapplication modules 10, communication action module 12 may, in someexamples, determine a contextually related operation to cause a mapsapplication to provide directions to be less likely. In some examples,communication action module 12 may determine operations without regardto what application modules are installed at computing device 2. In someexamples, contextually related operations determined by communicationaction module 12 may be or may include operations to obtain relevantapplications (e.g., from an application store or other source). Forinstance, if communication action module 12 determines that an operationto create a reservation at a restaurant is more likely, but applicationmodules 10 does not include any application configured to create suchreservations, communication action module 12 may also determine that anoperation to obtain (e.g., after obtaining permission from the user) anapplication module capable of creating the reservation is more likely.

In the example of FIG. 1, computing device 2 may execute the operation.For instance, communication action module 12 may cause computing device2 to perform the determined operation by executing a call of anapplication programming interface (API) of one of application modules10. In the example of FIG. 1, communication action module 12 maydetermine an operation to provide directions to Yotteru SushiRestaurant. Communication action module 12 may use an API to call one ofapplication modules 10 (e.g., application module 10B) to providedirections (e.g., in a turn-by-turn mode or otherwise) to the YotteruSushi Restaurant location. That is, in the example of FIG. 1,communication action module 12 may perform an API call to applicationmodule 10B to cause computing device 2 to provide directions to YotteruSushi Restaurant.

In some examples, computing device 2 may perform contextually relatedoperations without outputting information for display and/or without anyfurther user interaction. For instance, in some examples, a calendarapplication of computing device 2 may perform a contextually relatedoperation to create a calendar item without indicating to the user thatthe operation is being performed, without prompting the user to cancelor allow the operation, and/or without displaying any information aboutthe created item to the user.

In some examples, computing device 2 may display additional informationand/or prompt the user for acceptance as part of performing theoperation or prior to performing the operation. For instance, in theexample of FIG. 1, application module 10B may send data to UI module 6to cause UI device 4 to output GUI 30. As shown in the example of FIG.1, GUI 30 may indicate to the user of computing device 2 that computingdevice 2 is performing or about to perform the contextually relatedoperation. GUI 30 includes user selectable elements that may enable theuser to modify or cancel the operation, such as elements 34A and 34B. Insome examples, if computing device 2 does not receive an indication ofinput that selects one of elements 34A or 34B within a threshold amountof time (e.g., 1 second, 2 seconds, or other duration), computing device2 (e.g., application module 10B) may perform the operation. That is,application module 10B may provide directions to Yotteru SushiRestaurant (e.g., by causing UI device 4 to output a map, turn-by-turntext directions, etc.).

In some examples, GUI 30 may additionally or alternatively provide auser selectable element to affirmatively cause computing device 2 toperform the operation. That is, in some examples computing device 2 mayprovide a confirmation dialogue which the user may accept or reject.Such a confirmation dialogue may be useful in instances where theoperation was determined to be only slightly contextually related to thecorrespondence, or where the operation may have a large impact on theuser (e.g., removing a scheduled calendar event, placing a phone call,purchasing an application or other product, etc.).

By providing candidate responses and automatically performing anoperation contextually related to a received communication and aselected response to the received communication, techniques of thepresent disclosure may reduce the amount of time required for a user ofcomputing device 2 to input a response to a received communicationand/or to perform other operations based on the correspondence. Forinstance, wearable computing devices, such as watches or other devices,may be able to display candidate responses to a user that are based on areceived communication and automatically perform contextually relatedoperations, thereby enabling the user to quickly tap or otherwise selectone of the candidate responses instead of having to use voicerecognition or other means to input a response and cause the device toperform related operations.

FIG. 2 is a block diagram illustrating details of one example of acomputing device for performing contextually related operationsresponsive to sending a response to a received communication, inaccordance with one or more techniques of the present disclosure. FIG. 2is described below within the context of FIG. 1. FIG. 2 illustrates onlyone particular example of computing device 2, and many other exampledevices having more, fewer, or different components may also beconfigurable to perform operations in accordance with techniques of thepresent disclosure.

While displayed as part of a single device in the example of FIG. 2,components of computing device 2 may, in some examples, be locatedwithin and/or part of different devices. For instance, in some examples,some or all of the functionality of communication action module 14 maybe located at a server system or other computing device (e.g.,accessible by computing device 2 via a network). That is, in someexamples, techniques of the present disclosure may be performed andutilized by a single computing device while in other examples thetechniques may be performed and/or utilized at a remote computingsystem, such as a distributed or “cloud” computing system. In some suchexamples, computing device 2 may receive a communication and determinecandidate responses by providing at least a portion of the receivedcommunication to a remote device (e.g., a server system) and receiving aresponse that includes the candidate responses. Responsive to receivinga selection of one of the candidate responses, computing device 2 maydetermine an operation contextually related to the candidate response bysending at least an indication of the selected candidate response to aremote device and receiving a response that includes instructions thatare executable to perform the operation.

In some examples, computing device 2 may represent a wearable computingdevice. A wearable computing device may be any computing device wearableor otherwise attachable to a user, such as a smartwatch, a head-mountedcomputing device (e.g., a computing device incorporated into glasses, ahat, earphones, a contact lens or other like items), implantabledevices, or any other device a user may attach to his or her person.Some wearable computing devices, such as a smartwatch, may include oneor more input devices to receive various types of input from a user, oneor more output devices to provide audio, visual, tactile, or otheroutput to the user, one or more network interfaces to communicate withother computing devices, one or more sensors to obtain information,and/or other components.

In some examples, a wearable device may communicate with a mobile device(e.g., a smartphone) or other device to perform some operations. Forinstance, in some examples, part of the techniques of the presentdisclosure may be performed by a smartwatch, while part may be performedby one or more other computing devices (e.g., a smartphone that iswirelessly linked with the smartwatch, a server device with which thesmartphone may communicate, or other devices). In other words, whiledescribed herein as being performed by one or more components ofcomputing device 2, some of the components configurable to determinecandidate responses and/or contextually related operations may bedistributed among a plurality of computing devices in accordance withthe techniques described herein.

As shown in the example of FIG. 2, computing device 2 includes userinterface (UI) device 4, one or more processors 40, one or more inputdevices 42, one or more output devices 44, one or more communicationsunits 46, one or more sensors 48 and one or more storage devices 50.Storage devices 50 further include user interface (UI) module 6,suggestion module 12, application modules 10, and communication actionmodule 14. Communication action module 14, in the example of FIG. 2,includes response suggestion module 54, action prediction module 56, andcontext module 58.

Each of components 4, 40, 42, 44, 46, 48, and 50 may be interconnected(physically, communicatively, and/or operatively) for inter-componentcommunications. In the example of FIG. 2, components 4, 40, 42, 44, 46,48, and 50 may be coupled by one or more communications channels (COMM.CHANNELS) 52. In some examples, communications channels 52 may include asystem bus, network connection, inter-process communication datastructure, or any other channel for communicating data. Modules 6, 10,12, 54, 56, and 58 may also communicate information with one another aswell as with other components in computing device 2.

In the example of FIG. 2, one or more input devices 42 may be operableto receive input. Examples of input are tactile, audio, and video input.Input devices 42, in one example, include a presence-sensitive ortouch-sensitive display, a mouse, a keyboard, a voice responsive system,a video camera, a microphone or other audio sensor, or any other type ofdevice for detecting input from a human or machine.

One or more output devices 44 may be operable, in the example of FIG. 2,to generate output. Examples of output are tactile, audio, and videooutput. Output devices 44 of computing device 2, in some examples,include a presence-sensitive display, sound card, video graphics adaptercard, speaker, cathode ray tube (CRT) monitor, liquid crystal display(LCD), or any other type of device for generating output to a human ormachine.

In some examples, UI device 4 may include functionality of input devices42 and/or output devices 44. In the example of FIG. 2, for instance, UIdevice 4 may be or may include a presence-sensitive input device. Insome examples, a presence-sensitive input device may detect an object atand/or near the presence-sensitive input device. As one example range, apresence-sensitive input device may detect an object, such as a fingeror stylus that is within two inches or less of the presence-sensitiveinput device. In another example range, a presence-sensitive inputdevice may detect an object six inches or less from thepresence-sensitive input device, and other ranges are also possible. Thepresence-sensitive input device may determine a location (e.g., an (x,y)coordinate) of the presence-sensitive input device at which the objectwas detected. The presence-sensitive input device may determine thelocation selected by the input device using capacitive, inductive,and/or optical recognition techniques. In some examples,presence-sensitive input device provides output to a user using tactile,audio, or video stimuli as described with respect to output devices 44,and may be referred to as a presence-sensitive display.

While illustrated as an internal component of computing device 2, UIdevice 4 may also represent an external component that shares a datapath with computing device 2 for transmitting and/or receiving input andoutput. That is, in some examples, UI device 4 may represent a built-incomponent of computing device 2, located within and physically connectedto the external packaging of computing device 2 (e.g., a screen on amobile phone or wearable computing device). In some examples, UI device4 may represent an external component of computing device 2, locatedoutside and physically separated from the packaging of computing device2 (e.g., a monitor, a projector, or other display device that shares awired and/or wireless data path with computing device 2).

In the example of FIG. 2, one or more communication units 46 may beoperable to communicate with external devices via one or more networksby transmitting and/or receiving network signals on the one or morenetworks. For example, computing device 2 may use communication units 46to transmit and/or receive radio signals on a radio network such as acellular radio network. Likewise, communication units 46 may transmitand/or receive satellite signals on a satellite network such as a GPSnetwork. Examples of communication unit 46 include a network interfacecard (e.g. such as an Ethernet card), an optical transceiver, a radiofrequency transceiver, or any other type of device that can send and/orreceive information. Other examples of communication units 46 mayinclude Near-Field Communications (NFC) units, Bluetooth® radios, shortwave radios, cellular data radios, wireless network (e.g., Wi-Fi®)radios, as well as universal serial bus (USB) controllers.

One or more sensors 48 may, in the example of FIG. 2, be operable togenerate data for use by components of computing device 2. Sensors 48may include any device or component capable of obtaining data aboutcomputing device 2, data about an environment in which computing device2 is situated, data about a user of computing device 2, or other data.That is, any of sensors 48 may be hardware, firmware, software, or acombination thereof for obtaining information. Examples of sensors 48may include a GPS sensor or GPS radio, a position sensor, anaccelerometer or other motion sensor, a camera, a compass, amagnetometer, a light sensor, an infrared sensor, a microphone or otheraudio sensor, a radiation sensor, a temperature sensor, a barometer, analtimeter, or other data gathering components.

One or more storage devices 50 may be operable, in the example of FIG.2, to store information for processing during operation of computingdevice 2. In some examples, storage devices 50 may represent temporarymemory, meaning that a primary purpose of storage devices 50 is notlong-term storage. For instance, storage devices 50 of computing device2 may be volatile memory, configured for short-term storage ofinformation, and therefore not retain stored contents if powered off.Examples of volatile memories include random access memories (RAM),dynamic random access memories (DRAM), static random access memories(SRAM), and other forms of volatile memories known in the art.

Storage devices 50, in some examples, also represent one or morecomputer-readable storage media. That is, storage devices 50 may beconfigured to store larger amounts of information than a temporarymemory. For instance, storage devices 50 may include non-volatile memorythat retains information through power on/off cycles. Examples ofnon-volatile memories include magnetic hard discs, optical discs, floppydiscs, flash memories, or forms of electrically programmable memories(EPROM) or electrically erasable and programmable (EEPROM) memories. Inany case, storage devices 50 may, in the example of FIG. 2, storeprogram instructions and/or data associated with modules 6, 10, 12, 54,56, and 58.

In the example of FIG. 2, one or more processors 40 may be configured toimplement functionality and/or execute instructions within computingdevice 2. For example, processors 40 may be operable to receive andexecute instructions stored at storage devices 50 that implement thefunctionality of UI module 6, application modules 10, suggestion module12, response suggestion module 54, action prediction module 56, andcontext module 58. These instructions, executed by processors 40, maycause computing device 2 to store information within storage devices 50(e.g., temporary memories) during program execution. Processors 40 mayexecute instructions of modules 6, 10, 12, 54, 56, and 58 to cause UIdevice 4 to output candidate responses in response to computing device 2receiving a communication. Furthermore, processors 40 may execute theinstructions of modules 6, 10, 12, 54, 56, and 58 to determine anoperation that is contextually related to a response communication. Thatis, modules 6, 10, 12, 54, 56, and 58 may be operable by processors 40to perform various actions, including predicting candidate responses toa received communication and predicting one or more operations based onthe received communication and a selected response communication.

Computing device 2 may, in the example of FIG. 2, receive acommunication. For instance, one of communication units 46 may receivedata from a network (e.g., a wireless network or cellular network)adhering to the SMS protocol and representing a text message.Communications units 46 may provide the received data to one or more ofapplication modules 10 that are designated (e.g., previously designatedby a user) to handle data adhering to the SMS protocol, such asapplication module 10A.

Application module 10A may receive the text message and cause computingdevice 2 to output at least an indication of the text message. Forinstance, application module 10A may send information to UI module 6that causes UI device 4 or any of output devices 44 to display a visualrepresentation of the text (e.g., “We're all at Yotteru SushiRestaurant.”) included in the text message. The user of computing device2 may view the output and provide input that instructs computing device2 to respond to the text message. That is, UI module 6 may receive anindication of input, performed at UI device 4 or any of input devices 42that selects an option to respond to the received text message. UImodule 6 may provide the indication of input to application module 10A.

Responsive to receiving the input to instruct computing device 2 torespond to the text message, application module 10A may requestcandidate responses from communication action module 12. In requestingcandidate responses, application module 10A may provide information thatwas included in the text message (e.g., the text “We're all at YotteruSushi Restaurant.”) and/or other information (e.g., an indication of thesender of the text message). That is, a request for candidate responses,in various examples, may include information that was included in areceived email, a received text message, a received applicationnotification, a received calendar invite, a reminder, or othercommunication. In some examples, the request for candidate responses mayinclude all of the information included in the received communication.That is, the request may include the entire email, text message, etc. Inother examples, the request may include a portion of information. Forinstance, the request may include a date and time indicated in thereceived communication, an indicated location, the names of participantsin a meeting, emoji, a picture, or other information. Communicationaction module 12 may receive the request for candidate responses andprovide the request to response suggestion module 54.

Response suggestion module 54 may be operable to receive a request forcandidate responses and determine one or more candidate responses to acommunication. In some examples, response suggestion module 54 may beoperable to determine candidate responses by using natural languageprocessing or other techniques to parse the received communication,determine semantic meaning and/or one or more semantic elements, andpredict candidate responses that a user may be likely to select as aresponse to the received communication. For instance, responsesuggestion module 54 may parse text to obtain important phrases anddistill key ideas (e.g., semantic elements). For instance, responsesuggestion module 54 may implement computer learning techniques trainedfrom various corpora sources. That is, response suggestion module 54 maypredict candidate responses by using data that was previously learnedfrom parsing large volumes of text including indications (provided byhumans) of different parts of speech, indications of key ideas,indications of types of speech, and other indications. Responsesuggestion module 54 may learn from these corpora and apply the resultsto text input. For instance, response suggestion module 54 may determinevarious parts of speech (e.g., subject, object, verb, noun, noun phrase,etc.) for the text input and determine subjects or items having valueswithin a conceptual data space (e.g., a semantic space) for the input.The semantic elements may be topics or subjects that may be mentionedin, referred to, described in, or otherwise present in the input. Insome examples, response suggestion module 54 may determine semanticelements using clustering. While described with respect to responsesuggestion module 54, one or more of the aforementioned techniques maybe performed at one or more remote computing devices other thancomputing device 2.

After determining semantic elements for some text input (e.g., acommunication), response suggestion module 54 may use natural languageprocessing techniques to determine one or more responses that a usermight be likely to select. In some examples, response suggestion module54 may determine the responses based on previous responses selected bythe user and/or other users. That is, response suggestion module 54 mayuse various statistics to determine word choice, phrase construction,and other factors relevant to creating candidate responses. Bycondensing received communications to semantic elements (e.g., one ormore subjects), response suggestion module 54 may determine a semanticmeaning of the communication (e.g., what the communication is about). Insome examples, response suggestion module 54 may weigh semantic elementsby importance (e.g., based on predetermined importance values, based onfrequency of occurrence in a communication, and/or other factors). Byanalyzing previous responses to communications having substantiallysimilar semantic meanings (e.g., similar important subjects), responsesuggestion module 54 may increase the likelihood that the predictedcandidate responses will be relevant to the received communication. Twosemantic meanings may, in various examples, be substantially similar ifthey have a threshold number (e.g., one, three, five, or other number)of semantic elements that are the same, if they have a thresholdpercentage (e.g., twenty percent, fifty percent, or other percentage) ofsemantic elements the same, if one semantic meaning has a thresholdnumber of semantic elements that exist in the other semantic meaning, ifthe semantic meanings are probabilistically similar, or if some othercriteria are satisfied.

In the example of FIG. 2, response suggestion module 54 might determine,based on the received communication, that the sender of thecommunication is at a place called Yotteru Sushi Restaurant with one ormore other people. Based on previous responses provided by the user ofcomputing device 2 and/or other users, response suggestion module 54 maypredict responses that the user might select, such as responses toindicate that the user plans to join, responses to query where thelocation is, or other responses. In other words, response suggestionmodule 54 may, in some examples, use various machine learning techniquessuch as natural language processing to parse communications anddetermine candidate responses.

In some examples, response suggestion module 54 may be operable todetermine candidate responses to the received communication by using oneor more words and/or punctuation marks from the communication as inputfor one or more lookup tables. For instance, response suggestion module54 may determine a type of communication by searching a lookup table(e.g., a communication type lookup table) for the first word in eachsentence and the corresponding punctuation mark at the end of thesentence. The communication type lookup table may include entries thatindicate a type of communication for various combinations, such as a“location request” type that corresponds to the combination of “where”and a question mark character, a “time request” type that corresponds tothe combination of “when” and question mark character, a “confirmationrequest” type that corresponds to the combination of “want” and questionmark character, a “status update” type that corresponds to “we” (or“we're,” or “I,” or “I'm”) and a period character (or no punctuation),etc. Thus, for the text message, “We're all at Yotteru SushiRestaurant.” response suggestion module 54 may determine a “statusupdate” type.

Response suggestion module 54 may also determine one or more subjects ofthe received communication by using words or word combinations to searcha lookup table (e.g., a subject matter lookup table) that definescertain types of subjects, such as locations, people, objects or items,or other types. If a word is included in the subject matter lookuptable, the word may be a subject of the corresponding type. Forinstance, if “Restaurant” or “Yotteru Sushi Restaurant” is included inthe subject matter lookup table, it may correspond to a “location” typeand/or an “item” type. That is, a restaurant may correspond to aspecific location, a specific item (e.g., the restaurant itself), orboth.

In some examples, response suggestion module 54 may determine candidateresponses based on the determined communication type, the determinedsubject type, and the subject itself. For instance, response suggestionmodule 54 may access another lookup table (e.g., a response lookuptable) using these values as input to receive one or more candidateresponses as output. In the example of FIG. 2, the response lookup tablemay include an entry for a “status update” type with a “location” typesubject and a subject of “Yotteru Sushi Restaurant.” The entry maycorrespond to stored candidate responses such as “On my way,” “I can'tmake it,” “Sounds like fun,” or other candidate responses. In variousexamples, response suggestion module 54 may use more or fewer tables ina similar fashion to determine candidate responses. That is, whiledescribed in the example of FIG. 2 as a communication type lookup table,a subject matter lookup table and a response lookup table, techniques ofthe present disclosure may, in other examples, utilize any number oflookup tables providing additional or other specificity. Afterdetermining candidate responses, response suggestion module 54 mayprovide the candidate responses to application module 10A.

In the example of FIG. 2, application module 10A may receive thecandidate responses and send data to UI module 6 to cause UI device 4 orone or more of output devices 46 to display a representation of thecandidate responses. The user of computing device 2 may view thedisplayed responses and provide input that selects one of the candidateresponses or inputs a custom response. That is, UI module 6 may receivean indication of input, performed at UI device 4 or any of input devices42 that selects a response option representing a candidate response orthat inputs a custom response. UI module 6 may provide the indication ofinput to application module 10A.

Application module 10A may receive the indication of input that selectsa response and, responsive to receiving the indication, may send theselected response. For instance, in the example of FIG. 2, application10A may receive an indication of input to select the candidate response“On my way” and may send the selected candidate response (e.g., to thesender of the received communication).

In the example of FIG. 2, application module 10A may send an indicationof the selected candidate response to communication action module 12.Communication action module 12 may provide the indication of thecandidate response to action suggestion module 56 for determination ofone or more operations that are contextually related to the candidateresponse, based at least in part on at least one of the receivedcommunication and the candidate response. That is, action suggestionmodule 56 may be operable to determine at least one operation that isrelevant to the context of the selected candidate response based atleast in part on the communications.

In some examples, action suggestion module 56 may predict contextuallyrelated operations based on computer learning techniques and/or naturallanguage processing of the received communication and candidateresponse. For instance, action suggestion module 56 may parse theselected candidate response to determine a semantic meaning for thecandidate response (e.g., what the selected candidate response isabout). Along with the information indicating the semantic meaning ofthe received communication, action suggestion module 56 may use thesemantic meaning of the selected candidate response to predict anoperation that the user may be likely to cause computing device 2 toperform when computing device 2 has received communications and sentresponses that were associated with substantially similar semanticmeanings. For instance, action suggestion module 56 may determineprevious correspondences (e.g., previously received communications andpreviously sent responses) that were associated with a group of semanticelements that included a threshold number of semantic elements existentin a group of semantic elements associated with the selected candidateresponse and/or the received communication. Action suggestion module 56may determine operations that were performed within a threshold amountof time after the determined previous correspondences. For instance,action suggestion module 56 may determine operations that were performedimmediately after the previous correspondence, within 2 minutes of thecorrespondence, within 5 minutes, or otherwise performed at a similartime. That is, in some examples, action suggestion module 56 may predictcontextually related operations by learning what operations computingdevices (e.g., computing device 2 and/or other computing devices) havepreviously performed, given substantially similar previously receivedcommunications and corresponding responses.

In the example of FIG. 2, for instance, computing devices that receiveda status update message indicating a location and sent a response thatindicated the user intends to travel to the location may have receivedinput that caused the computing device to provide directions to thelocation. Based on this correlation, action suggestion module 56 maydetermine an operation to obtain driving directions to Yotteru SushiRestaurant.

In some examples, action suggestion module 56 may determine contextuallyrelated operations using other methods. For instance, action suggestionmodule 56 may parse the selected candidate response using one or morelookup tables to determine a response type, and a subject type. Actionsuggestion module 56 may then use the communication type, subject,and/or subject type for the received communication, along with theresponse type, subject, and/or subject type of the selected candidateresponse to look up a corresponding operation that is contextuallyrelated to the candidate response.

In some examples, action suggestion module 56 may parse the selectedcandidate response and/or the received communication and determine aplurality of operations based on one or more semantic elements includedin the communications. A semantic element, generally, may be anyspecified sequence of symbols or combinations of symbols that havemeaning in a spoken and/or written language. Examples of semanticelements may include one or more words (e.g., a noun, a noun phrase, averb, a verb phrase, etc.), one or more numbers or number groups, one ormore emoticons, or a combination of such items. Action suggestion module56 may determine operations by searching the received communicationand/or the selected candidate response for semantic elements associatedwith operations. If the communications contain any semantic elementassociated with a particular operation, action suggestion module 56 mayinclude the particular operation in the plurality of operations. Actionsuggestion module 56 may determine respective scores associated witheach of the plurality of operations. For instance, a respective scorefor a particular operation may indicate the number of semantic elementsincluded in (or otherwise associated with) the communications that arealso associated with the particular operation. That is, actionsuggestion module 56 may compare each semantic element included in thecommunications with each semantic element associated with the operation.If the semantic elements are the same (or at least substantiallysimilar), action suggestion module 56 may increment the correspondingscore. Two semantic elements may be substantially similar if theelements are the same, or if the elements are probabilistically within adegree of similarity that is greater than a threshold value. Actionsuggestion module 56 may then select, as the operation that iscontextually related to the candidate response, the operation associatedwith the best respective score. In other words, action suggestion module56 may, in some examples, use groups of semantic elements associatedwith operations to determine which operation has the most semanticelements in common with the communications.

In some examples, action suggestion module 56 may determine theoperation that is contextually related to the selected candidateresponse by parsing the selected candidate response to determine asemantic meaning of the candidate response. Action suggestion module 56may use the semantic meaning of the candidate response and/or thesemantic meaning of the received communication to determine one ofapplication modules 10 that is registered to handle informationassociated with the determined semantic meaning or meanings. Anapplication module may be registered to handle information associatedwith a semantic meaning by an operating system of the computing device,a user of the computing device, an administrator of the computingdevice, or registered in some other way. The registered applicationmodule may be operable to receive information associated with thesemantic meaning (e.g., information of a specific type, informationabout specific topics, etc.) and act on the information. For instance,action suggestion module 56 may determine that the semantic meaning ofthe received communication and/or the selected candidate response is acontact of the user of computing device 2 by the name of “George.”Action suggestion module 56 may determine that the applicationregistered to handle contacts is an address book application. Based onthe determination of the registered application, action suggestionmodule 56 may determine an operation to cause the registered applicationto execute on information included in at least one of the receivedcommunication and/or the selected candidate response. For instance,action suggestion module 56 may determine an operation to cause theaddress book application to execute with the information “George” (e.g.,which may cause the address book application to search for contacts nameGeorge).

In the example of FIG. 2, action suggestion module 56 may execute thedetermined operation. For instance, action suggestion module 56 may usean API call of one of application modules 10 (e.g., a maps applicationor navigation application) and include an indication of the location(e.g., Yotteru Sushi Restaurant) to cause computing device 2 to executethe operation and provide directions to Yotteru Sushi Restaurant. Actionsuggestion module 56 and/or application modules 10 may provide data toUI module 6 that causes UI 4 or one or more of output devices 44 tooutput the information for display.

In some examples, determining candidate responses and/or determiningcontextually related operations may be based additionally oralternatively on contextual information that defines a current contextof computing device 2 and/or a current context of a user of computingdevice 2 (e.g., as defined by items of contextual information). That is,in some examples, response suggestion module 54 may determine candidateresponses based on the current context of computing device 2 and/or theuser. Furthermore, action suggestion module 56 may, in some examples,determine operations that are contextually related to the candidateresponse based on the current context of computing device 2 and/or theuser. Response suggestion module 54 and/or action suggestion module 56may communicate with context module 58 to obtain contextual informationthat defines the current context.

Context module 58 may be operable to obtain contextual information foruse in determining candidate responses and/or contextually relatedoperations. The information may include contextual information aboutcomputing device 2. For instance, context module 58 may obtain items ofinformation representing one or more locations of computing device 2,information indicating a time and/or time zone setting of computingdevice 2, information indicating movement of computing device 2, anambient noise level of the environment around computing device 2, anambient light level of the environment, or other sensor data (e.g.,obtained from sensors 48). Additional examples of items of contextualinformation may include information indicating one or more applicationscurrently being executed by computing device 2 and/or an activity inwhich a currently executing application is engaging, informationindicating the type of the received communication (e.g., email, textmessage, calendar alert, or other type), information indicating the typeof response that the computing device will be sending, informationindicating the time at which the communication was received, informationindicating previous communications to or from the sender, informationindicating an occupation of the user, information indicating an activityin which the user is currently engaged, information indicating thesender of the received communication, or information indicating one ormore recipients of the response that the computing device will besending.

In some examples, context module 58 may additionally or alternativelyobtain items of contextual information about a user of computing device2. For instance, context module 58 may obtain contextual informationabout the user from an email account associated with a user, a socialnetwork service account of the user, a web search history associatedwith the user, a location history associated with the user, or any othersource. That is, in some examples, context module 58 may obtain anyinformation usable by one or more components of communication actionmodule 12 to determine more accurate candidate responses and/or moreaccurate operations that are contextually related to candidateresponses.

Context module 58 may only obtain contextual information about computingdevice 2 and/or a user of computing device 2 if the user providesexplicit permission. That is, context module 58 may only obtain varioussources of information for predicting candidate responses and/orcontextual operations if a user of computing device 2 explicitly allowscommunication action module 12 to access various sources of contextualinformation. For instance, computing device 2 may prompt the user toallow access to sources of contextual information such as sensors 48, anemail account of the user, a social network service account, or otheraccount. If the user explicitly agrees to allow access, one or morecomponents of communication action module 12 may obtain the contextualinformation and use the contextual information in determining candidateresponses and/or determining contextually related operations.

In the example of FIG. 2, context module 58 may obtain contextualinformation (e.g., from one of sensors 48) indicating that computingdevice 2 is currently at Location X. Context module 58 may also obtaincontextual information (e.g., from a maps application) indicating aplurality of locations of Yotteru Sushi Restaurant. That is, contextmodule 58 may obtain information indicating a location of computingdevice 2 and information indicating locations of Yotteru Sushi.

In some examples, context module 58 may be operable to determine acurrent context of computing device 2 based on obtained items ofcontextual information. For instance, context module 58 may define acurrent context as a collection of all obtained items of contextualinformation. In another example, contextual module 58 may define acurrent context by determining weights for various items of contextualinformation. Context module 58 may weigh each item of contextualinformation based on a predicted accuracy and/or a predicted importanceof the item. In the example of FIG. 2, context module 58 may weigh thecontextual information indicating Location X very heavily because thelocation of computing device 2 is likely to be very accurate. Contextmodule 58 may weigh the contextual information indicating each of thelocations of Yotteru Sushi Restaurant lighter, because the text “YotteruSushi Restaurant” is a general location identifier and it is not certainwhich of the restaurant locations the received communication wasreferring to. By determining the current context, context module 58 mayallow response suggestion module 54 and/or action suggestion module 56to determine more accurate candidate responses that are more likely tobe selected by a user and/or more closely related operations. Forinstance, response suggestion module 54 may, in the example of FIG. 2,use the determined current context to determine the candidate response“The one on Washington St.?” That is, response suggestion module 54 mayreceive the context from context module 58 including contextualinformation that indicates the multiple locations of Yotteru SushiRestaurant and the associated weights. Because each restaurant isweighted approximately the same, response suggestion module maydetermine that it is not certain which of the Yotteru Sushi Restaurantsthe received communication is referring to.

In various examples, response suggestion module 54 may use thedetermined current context in addition to or instead of otherinformation (e.g., information included in a received communication,metadata about the received communication, etc.) to determine candidateresponses. That is, in accordance with the techniques described herein,response suggestion module 54 may, in some examples, determine one ormore candidate responses that are associated with a semantic meaningthat is similar to the semantic meaning of the received communication(e.g., based on the natural language processing of previously receivedcommunications, previously selected candidate responses, and/orpreviously input custom responses). Response suggestion module 54 may,in some examples, rank the determined candidate responses by theircontextual similarity to the current context of computing device 2.Contextual similarity may be a measure of how many items of contextualinformation are the same between two contexts. For instance, a candidateresponse may be associated with a context that is substantially similarto the current context if one of the contexts includes a thresholdnumber (e.g., one, three, five, or other number) of items of contextualinformation that exist in the other context. Other examples ofdetermining substantial similarity between contexts include determiningwhether the contexts each include a threshold percentage of items ofcontextual information that are the same (e.g., twenty percent, fiftypercent, or other percentage), or other methods of measuring thecontextual similarity.

Response suggestion module 54 may output the determined candidateresponses that are associated with contexts or items of contextualinformation that are the most similar to the current context or thecurrent items of contextual information. For instance, responsesuggestion module 54 may score the determined candidate responses basedon the number of items of contextual information that are the same asitems of contextual information included in the current context.Response suggestion module 54 may then output the determined candidateresponses that are associated with the highest scores. In some examples,response suggestion module 54 may output the top responses (e.g., thetop three, the top five, etc.). In some examples, response suggestionmodule 54 may output responses associated with scores that are above athreshold score.

Furthermore, action suggestion module 56 may, in various examples, usethe determined current context in addition to or instead of otherinformation (e.g., information included in the communications, metadata,other communications, etc.) to determine contextually relatedoperations. That is, in accordance with the techniques described herein,action suggestion module 56 may determine one or more operations thatare associated with a semantic meaning that is similar to the semanticmeaning that is associated with the received communication and/or theselected candidate response (e.g., based on natural language processingof previously received communications, previously selected responses,and/or previous operations). In some examples, action suggestion module56 may rank the determined operations by their contextual similarity tothe current context of computing device 2. Action suggestion module 56may output the determined operations that are associated with contextsor items of contextual information that are the most similar to thecurrent context or the current items of contextual information. Forinstance, action suggestion module 56 may score the determinedoperations based on the number of items of contextual information thatare the same as items of contextual information included in the currentcontext. Action suggestion module 56 may then output the determinedoperations that are associated with the highest score or scores. In someexamples, action suggestion module 56 may output the top operations(e.g., the top operation, the top three operations, etc.). In someexamples, action suggestion module 56 may output operations associatedwith scores that are above a threshold score.

FIG. 3 is a block diagram illustrating an example computing device thatoutputs graphical content for display at a remote device, in accordancewith one or more techniques of the present disclosure. Graphicalcontent, generally, may include any visual information that may beoutput for display, such as text, images, a group of moving images, etc.The example shown in FIG. 3 includes a computing device 60,presence-sensitive display 64, communication unit 70, projector 80,projector screen 82, mobile device 86, and visual display device 90.Although shown for purposes of example in FIGS. 1 and 2 as a stand-alonecomputing device 2, a computing device such as computing device 60 may,generally, be any component or system that includes a processor or othersuitable computing environment for executing software instructions and,for example, need not include a presence-sensitive display.

As shown in the example of FIG. 3, computing device 60 may be aprocessor that includes functionality as described with respect toprocessors 40 in FIG. 2. In such examples, computing device 60 may beoperatively coupled to presence-sensitive display 64 by a communicationchannel 62A, which may be a system bus or other suitable connection.Computing device 60 may also be operatively coupled to communicationunit 70, further described below, by a communication channel 62B, whichmay also be a system bus or other suitable connection. Although shownseparately as an example in FIG. 3, computing device 60 may beoperatively coupled to presence-sensitive display 64 and communicationunit 70 by any number of one or more communication channels.

In other examples, such as illustrated previously by computing device 2in FIGS. 1-2, a computing device may refer to a portable or mobiledevice such as mobile phones (including smart phones), laptop computers,etc. In some examples, a computing device may be a desktop computer,tablet computer, smart television platform, camera, personal digitalassistant (PDA), server, mainframe, etc.

Presence-sensitive display 64, as one of input devices 42 and/or outputdevices 44 as shown in FIG. 2, may include display device 66 andpresence-sensitive input device 68. Display device 66 may, for example,receive data from computing device 60 and display the graphical content.In some examples, presence-sensitive input device 68 may determine oneor more user inputs (e.g., continuous gestures, multi-touch gestures,single-touch gestures, etc.) at presence-sensitive display 64 usingcapacitive, inductive, and/or optical recognition techniques and sendindications of such user input to computing device 60 usingcommunication channel 62A. In some examples, presence-sensitive inputdevice 68 may be physically positioned on top of display device 66 suchthat, when a user positions an input unit over a graphical elementdisplayed by display device 66, the location at which presence-sensitiveinput device 68 corresponds to the location of display device 66 atwhich the graphical element is displayed. In other examples,presence-sensitive input device 68 may be positioned physically apartfrom display device 66, and locations of presence-sensitive input device68 may correspond to locations of display device 66, such that input canbe made at presence-sensitive input device 68 for interacting withgraphical elements displayed at corresponding locations of displaydevice 66.

As shown in FIG. 3, computing device 60 may also include and/or beoperatively coupled with communication unit 70. Communication unit 70may include functionality of communication units 42 as described in FIG.2. Examples of communication unit 70 may include a network interfacecard, an Ethernet card, an optical transceiver, a radio frequencytransceiver, or any other type of device that can send and receiveinformation. Other examples of such communication units may includeBluetooth, 3G, and WiFi radios, Universal Serial Bus (USB) interfaces,etc. Computing device 60 may also include and/or be operatively coupledwith one or more other devices, e.g., input devices, output devices,memory, storage devices, etc. that are not shown in FIG. 3 for purposesof brevity and illustration.

FIG. 3 also illustrates a projector 80 and projector screen 82. Othersuch examples of projection devices may include electronic whiteboards,holographic display devices, and any other suitable devices fordisplaying graphical content. Projector 80 and projector screen 82 mayinclude one or more communication units that enable the respectivedevices to communicate with computing device 60. In some examples, theone or more communication units may enable communication betweenprojector 80 and projector screen 82. Projector 80 may receive data fromcomputing device 60 that includes graphical content. Projector 80, inresponse to receiving the data, may project the graphical content ontoprojector screen 82. In some examples, projector 80 may determine one ormore user inputs (e.g., continuous gestures, multi-touch gestures,single-touch gestures, etc.) at projector screen using opticalrecognition or other suitable techniques and send indications of suchuser input using one or more communication units to computing device 60.In such examples, projector screen 82 may be unnecessary, and projector80 may project graphical content on any suitable medium and detect oneor more user inputs using optical recognition or other such suitabletechniques.

Projector screen 82, in some examples, may include a presence-sensitivedisplay 84. Presence-sensitive display 84 may include a subset offunctionality or all of the functionality of input devices 4 and/oroutput devices 6 as described in this disclosure. In some examples,presence-sensitive display 84 may include additional functionality.Projector screen 82 (e.g., an electronic whiteboard), may receive datafrom computing device 60 and display the graphical content. In someexamples, presence-sensitive display 84 may determine one or more userinputs (e.g., continuous gestures, multi-touch gestures, single-touchgestures, etc.) at projector screen 82 using capacitive, inductive,and/or optical recognition techniques and send indications of such userinput using one or more communication units to computing device 60.

FIG. 3 also illustrates mobile device 86 and visual display device 90.Mobile device 86 and visual display device 90 may each include computingand connectivity capabilities. Examples of mobile device 86 may includee-reader devices, convertible notebook devices, hybrid slate devices,etc. Examples of visual display device 90 may include othersemi-stationary devices such as televisions, computer monitors, etc. Asshown in FIG. 3, mobile device 86 may include a presence-sensitivedisplay 88. Visual display device 90 may include a presence-sensitivedisplay 92. Presence-sensitive displays 88, 92 may include a subset offunctionality or all of the functionality of input devices 42 and/oroutput devices 44 as described in this disclosure. In some examples,presence-sensitive displays 88, 92 may include additional functionality.In any case, presence-sensitive display 92, for example, may receivedata from computing device 60 and display the graphical content. In someexamples, presence-sensitive display 92 may determine one or more userinputs (e.g., continuous gestures, multi-touch gestures, single-touchgestures, etc.) at projector screen using capacitive, inductive, and/oroptical recognition techniques and send indications of such user inputusing one or more communication units to computing device 60.

As described above, in some examples, computing device 60 may outputgraphical content for display at presence-sensitive display 64 that iscoupled to computing device 60 by a system bus or other suitablecommunication channel. Computing device 60 may also output graphicalcontent for display at one or more remote devices, such as projector 80,projector screen 82, mobile device 86, and visual display device 90. Forinstance, computing device 60 may execute one or more instructions togenerate and/or modify graphical content in accordance with techniquesof the present disclosure. Computing device 60 may output the data thatincludes the graphical content to a communication unit of computingdevice 60, such as communication unit 70. Communication unit 70 may sendthe data to one or more of the remote devices, such as projector 80,projector screen 82, mobile device 86, and/or visual display device 90.In this way, computing device 60 may output the graphical content fordisplay at one or more of the remote devices. In some examples, one ormore of the remote devices may output the graphical content at apresence-sensitive display that is included in and/or operativelycoupled to the respective remote devices.

In some examples, computing device 60 may not output graphical contentat presence-sensitive display 64 that is operatively coupled tocomputing device 60. In other examples, computing device 60 may outputgraphical content for display at both a presence-sensitive display 64that is coupled to computing device 60 by communication channel 62A, andat one or more remote devices. In such examples, the graphical contentmay be displayed substantially contemporaneously at each respectivedevice. For instance, some delay may be introduced by the communicationlatency to send the data that includes the graphical content to theremote device. In some examples, graphical content generated bycomputing device 60 and output for display at presence-sensitive display64 may be different than graphical content display output for display atone or more remote devices.

Computing device 60 may send and receive data using any suitablecommunication techniques. For example, computing device 60 may beoperatively coupled to external network 74 using network link 72A. Eachof the remote devices illustrated in FIG. 3 may be operatively coupledto network external network 74 by one of respective network links 72B,72C, and 72D. External network 74 may include network hubs, networkswitches, network routers, etc., that are operatively inter-coupledthereby providing for the exchange of information between computingdevice 60 and the remote devices illustrated in FIG. 3. In someexamples, network links 72A-72D may be Ethernet, ATM or other networkconnections. Such connections may be wireless and/or wired connections.

In some examples, computing device 60 may be operatively coupled to oneor more of the remote devices included in FIG. 3 using direct devicecommunication 78. Direct device communication 78 may includecommunications through which computing device 60 sends and receives datadirectly with a remote device, using wired or wireless communication.That is, in some examples of direct device communication 78, data sentby computing device 60 may not be forwarded by one or more additionaldevices before being received at the remote device, and vice-versa.Examples of direct device communication 78 may include Bluetooth,Near-Field Communication, Universal Serial Bus, WiFi, infrared, etc. Oneor more of the remote devices illustrated in FIG. 3 may be operativelycoupled with computing device 60 by communication links 76A-76D. In someexamples, communication links 76A-76D may be connections usingBluetooth, Near-Field Communication, Universal Serial Bus, infrared,etc. Such connections may be wireless and/or wired connections.

In accordance with techniques of the disclosure, computing device 60 maybe operatively coupled to visual display device 90 using externalnetwork 74. Computing device 60 may output graphical content for displayat presence-sensitive display 92. For instance, computing device 60 maysend data that includes a GUI for one or more of application modules 10to communication unit 70. Communication unit 70 may send the data thatincludes the GUI to visual display device 90 using external network 74.Visual display device 90, in response to receiving the data usingexternal network 74, may cause presence-sensitive display 92 to outputthe GUI. In response to receiving one or more user inputs, such as agesture at presence-sensitive display 92 (e.g., at a region ofpresence-sensitive display 92), visual display device 90 and other ofinput devices 4 may send indications of the inputs to computing device60 using external network 74. Communication unit 70 of may receive theindications of the inputs, and send the indications to computing device60.

Computing device 60 may, in the example of FIG. 3, receive acommunication. Computing device 60 may output an indication of thereceived communication (e.g., for display). For instance, computingdevice 60 may send data for a visual representation of the receivedcommunication to communication unit 70. Communication unit 70 may sendthe data to visual display device 90 via external network 74. Visualdisplay device 90 may cause presence-sensitive display 92 to output thevisual representation of the received communication for display.

Responsive to receiving an indication of user input to respond to thereceived communication (e.g., input performed at one of input devices42), computing device 60 (e.g., communication action module 12) maydetermine one or more candidate responses. Computing device 60 may senddata for a visual representation of at least one of the one or morecandidate responses to communication unit 70 for display.

Computing device 60 may receive input that selects a candidate responsefrom the one or more candidate responses and may send the candidateresponse (e.g., in response to the received communication). Computingdevice 60 may determine an operation that is contextually related to thecandidate response and execute the operation. In some examples, as partof performing the operation, computing device 60 may output a GUI fordisplay. That is, computing device 60 may send data for a visualinterface to communication unit 70. Communication unit 70 may send thedata to visual display device 90 via external network 74. Visual displaydevice 90 may cause presence-sensitive display 92 to output the visualinterface (e.g., to a user).

FIG. 4 is a block diagram illustrating an example computing device 2 andGUIs 100, 110, and 120 for performing contextually related operationsresponsive to sending a response to a received communication, inaccordance with one or more techniques of the present disclosure. Forpurposes of illustration only, the example of FIG. 4 is described belowwithin the context of FIGS. 1 and 2. As shown in the example of FIG. 4,computing device 2 includes UI device 4, UI module 6, applicationmodules 10, and communication action module 12. The components ofcomputing device 2 may, in the example of FIG. 4, have functionalitythat is the same as or similar to that described with respect to FIGS. 1and 2.

In some examples, performing one or more operations that arecontextually related to a selected candidate response may includeperforming one or more operations to obtain and/or output informationabout a subject of the selected candidate response and/or the receivedcommunication. That is, responsive to receiving input (e.g., from auser) to select a response to a received communication, a computingdevice implementing techniques of the present disclosure may, in someexamples, determine an operation to provide a user with informationabout a subject (e.g., a person, a location, an item or object, etc.)included in the communications. Examples of information about a subjectincluded in the communications may include weather information about alocation, travel information to a location (e.g., turn-by-turn drivingdirections, public transportation directions, available flights to thelocation, or other travel information), reviews of a location or object(e.g., a restaurant, a store, a movie, etc.) such as user reviews fromsocial media network services or critic reviews, biographicalinformation about a person, social media network service accounts for asubject or information obtained therefrom, recent news articles or mediastories about a subject, availability of an item or object for purchase(e.g., from one or more stores, shopping or auction websites orelsewhere), or other information about the subject that may be useful toa user. In other words, determining an operation that is contextuallyrelated to a candidate response may, in some examples, includedetermining an operation to obtain information about a subject includedin the received communication and/or the candidate response.

In accordance with the techniques described herein, computing device 2may receive a communication, such as an instant message, that includesthe text, “When are you going to New York?” One or more components ofcomputing device 2 (e.g., one of application modules 10) may cause UIdevice 4 to output GUI 100 for display. For instance, in the example ofFIG. 4, computing device 2 may provide the received communication to oneof application modules 10 that is designated to handle instant messages(e.g., application module 10B). Application module 10B may sendinformation to UI module 6 to cause GUI 100 to be output for display atUI device 4. In some examples, GUI 100 may be output in response toreceiving the instant message. In some examples, GUI 100 may be outputin response to input provided by a user instructing computing device 2to display the instant message.

In the example of FIG. 4, GUI 100 includes text 102 and user-selectableelement 104. Text 102 may be a representation of at least a portion ofthe information included in the instant message. In some examples, text102 may represent all the information included in the instant message.In some examples, the instant message may include additional or otherinformation. Element 104 may be an object of the GUI that a user mayselect to provide input instructing computing device 2 to respond to theinstant message.

Computing device 2 may, in the example of FIG. 4, receive input (e.g.,selection 106) selecting element 104. That is, UI device 4 may receiveselection 106 and send data representing the input to UI module 6. UImodule 6 may receive the data and provide an indication of the input toapplication module 10B. Application module 10B may determine that theinput corresponds to a selection of element 104. Consequently,application module 10B may request candidate responses fromcommunication action module 12.

In the example of FIG. 4, communication action module 12 may receive therequest and determine candidate responses to the instant message. Inorder to determine candidate responses, communication action module 12may parse the information included in the instant message and search forcombinations of words and/or punctuation marks using one or more lookuptables to determine a semantic meaning of the instant message (e.g.,what the instant message is about). For instance, Communication actionmodule 12 may determine that the word “when” and the punctuation mark“?” together indicate a request for a time value. Communication actionmodule 12 may also determine that the words “going to” represent anaction and that that the words “you” and “New York” represent subjects.

Communication action module 12 may, in the example of FIG. 4, obtainvarious types of contextual information in order to determine candidateresponses. That is, communication action module 12 may obtaininformation about computing device 2 and/or about the user of computingdevice 2. Based on the information, communication action module 12 maydetermine that the user is flying to New York City next Wednesday. Forinstance, communication action module 12 may obtain contextualinformation from an email account of the user that indicates a purchaseof a ticket to New York leaving at 5:23 PM on Apr. 23, 2014 for a personnamed Bryan Dons. Communication action module 12 may obtain contextualinformation from computing device 2 indicating that the user's name isBryan Dons. Communication action module 12 may also obtain contextualinformation from a clock of computing device 2 indicating that thecurrent date is Apr. 17, 2014. Based on this information, communicationaction module 12 may determine that the user is travelling to New Yorkby flight at approximately 5 PM next week, Wednesday. Based on theinformation included in the instant message and the obtained contextualinformation, communication action module 12 may determine one or morecandidate responses and send the candidate responses to applicationmodule 10B.

Application module 10B may receive the candidate responses and sendinformation to UI module 6 to cause UI device 4 to output GUI 110 fordisplay. As shown in the example of FIG. 4, GUI 110 includes responseoptions 112A, 112B, and 112C (collectively, “response options 112”).Each of responses options 112 may correspond to a respective candidateresponse. In the example of FIG. 4, the candidate responses determinedby communication action module 12 may be based on varying levels ofprivacy. That is, the candidate response corresponding to responseoption 112A indicates only a general response, while the candidateresponses corresponding to response options 112B and 112C may beincreasingly more specific responses. This may allow the user to selecthis or her desired level of privacy for the interaction. That is, insome examples, communication action module 12 may determine candidateresponses that provide varying levels of privacy to the user, therebyallowing the user to select how much or how little of his or herinformation he or she would like to share in a response.

In the example of FIG. 4, computing device 2 may receive an indicationof user input (e.g., selection 114) that selects response option 112A(e.g., corresponding to the candidate response, “Next week”). That is,UI device 4 may receive selection 114 and send data representing theinput to UI module 6. UI module 6 may receive the data and provide anindication of the input to application module 10B. Application module10B may determine that the input corresponds to a selection of responseoption 112A.

Responsive to receiving the indication of user input that selectsresponse option 112A, application module 10B may cause computing device2 to send the candidate response “Next week” as an instant message. Thatis, responsive to receiving the indication of user input that selects acandidate response, computing device 2 may send the candidate response.Application module 10B may also send an indication of the candidateresponse to communication action module 12.

Communication action module 12 may, in the example of FIG. 4, receivethe indication of the candidate response and determine one or moreoperations to obtain information about a subject or subjects included inthe instant message and/or the selected candidate response. Forinstance, communication action module 12 may parse the language of theselected candidate response using lookup tables, in a fashion similar tothe parsing of the received communication. Consequently, communicationaction module 12 may determine a relative time value, “next week” fromthe candidate response. Communication action module 12 may use therelative time value, along with the subjects “New York” and “You,” andthe action “going to” as input for a lookup table that storescontextually relevant operations. The input values, together, maycorrespond to an operation to provide weather information about thelocation. That is, the lookup table may indicate that an affirmativeaction of travel (e.g., “going to”), a geographic location (e.g., “NewYork”), and a relative time (e.g., “next week”), together, correspond toa contextually related operation to provide weather information aboutthe location.

In the example of FIG. 4, communication action module 12 may execute theoperation. For instance, communication action module 12 may call an APIof a weather application of computing device 2, the call providing therelevant location and time frame (e.g., New York, N.Y. and next week).The weather application may execute and send data to UI module 6 thatcauses UI device 4 to output GUI 120. As shown in the example of FIG. 4,GUI 120 shows the weather for New York, N.Y. next week.

While described in the example of FIG. 4 as providing weatherinformation about a subject included in the received communicationand/or selected response, contextually related operations may providevarious other types of information about communication subjects. Forinstance, if communication action module 12 determines a location of NewYork, a relative time of next week, and an action of “want to go,”communication action module 12 may determine an operation to providetravel options to New York City. As another example, if communicationaction module 12 determines an action of “want to go,” and a “requestfor location” (e.g., based on the word “where” and a question markcharacter in a received communication), and determines a location of“Yotteru Sushi Restaurant” (e.g., entered by a user as a customresponse), communication action module 12 may determine an operation toprovide restaurant reviews of Yotteru Sushi Restaurant. Various othertypes of information may be provided in a number of examples, inaccordance with the techniques of the present disclosure.

FIG. 5 is a block diagram illustrating an example computing device 2 andGUIs 150, 160, and 170 for performing contextually related operationsresponsive to sending a response to a received communication, inaccordance with one or more techniques of the present disclosure. Forpurposes of illustration only, the example of FIG. 5 is described belowwithin the context of FIGS. 1 and 2. As shown in the example of FIG. 5,computing device 2 includes UI device 4, UI module 6, applicationmodules 10, and communication action module 12. The components ofcomputing device 2 may, in the example of FIG. 5, have functionalitythat is the same as or similar to that described with respect to FIGS. 1and 2.

In some examples, performing one or more operations that arecontextually related to a selected candidate response may includeperforming one or more operations to create, modify, or removescheduling items. That is, responsive to receiving input (e.g., from auser) to select a response to a received communication, a computingdevice implementing techniques of the present disclosure may, in someexamples, determine an operation to create, modify, or remove schedulingitems associated with a subject (e.g., a person, a location, an item orobject, etc.) included in the communications. Examples of operationsthat create, modify or remove scheduling items may include operations tocreate, modify, or remove a calendar event on a calendar associated withthe user, operations to create, modify, or remove an event on a socialmedia network service using an account associated with the user,operations to create, modify, or remove a reservation (e.g., a tablereservation at a restaurant, a hotel reservation, a rental reservation,a reservation for a plane flight, bus trip, or movie, a reservation fora meeting room, etc.), operations to create, modify, or remove a serviceappointment, or other operation to address a scheduling item that may beuseful to a user. In other words, determining an operation that iscontextually related to a candidate response may, in some examples,include determining an operation to create a scheduling item, modify ascheduling item, or remove a scheduling item that relates to thereceived communication and/or the selected response.

In accordance with the techniques described herein, computing device 2may receive a communication, such as an email, that includes the text,“Want to go to Yotteru Sushi Restaurant at 7 tomorrow night?” One ormore components of computing device 2 (e.g., one of application modules10) may cause UI device 4 to output GUI 150 for display. For instance,in the example of FIG. 5, computing device 2 may provide the receivedcommunication to one of application modules 10 that is designated tohandle email messages (e.g., application module 10C). Application module10C may send information to UI module 6 to cause GUI 150 to be outputfor display at UI device 4. In some examples, GUI 150 may be output inresponse to receiving the email. In some examples, GUI 150 may be outputin response to input provided by a user instructing computing device 2to display the email.

In the example of FIG. 5, GUI 150 includes text 152 and user-selectableelement 154. Text 152 may be a representation of at least a portion ofthe information included in the email. In some examples, text 152 mayrepresent all the information included in the email. In some examples,the email may include additional or other information, such as headerinformation. Element 154 may be an object of the GUI that a user mayselect to provide input instructing computing device 2 to respond to theemail.

Computing device 2 may, in the example of FIG. 5, receive input (e.g.,selection 156) selecting element 154. That is, UI device 4 may receiveselection 156 and send data representing the input to UI module 6. UImodule 6 may receive the data and provide an indication of the input toapplication module 10C. Application module 10C may determine that theinput corresponds to a selection of element 154. Consequently,application module 10C may request candidate responses fromcommunication action module 12.

In the example of FIG. 5, communication action module 12 may receive therequest and determine candidate responses to the email. In order todetermine candidate responses, communication action module 12 may parsethe information included in the email using natural language processingor other language processing techniques to determine a semantic meaningfor the instant message (e.g., what the instant message is about). Forinstance, Communication action module 12 may determine that the instantmessage is requesting information from the user about an action the useris taking with respect to a location, New York.

Communication action module 12 may, in the example of FIG. 5, obtainvarious types of contextual information in order to determine candidateresponses. That is, communication action module 12 may obtaininformation about computing device 2 and/or about the user of computingdevice 2. Based on the information, communication action module 12 maydetermine that the user has a meeting until 7 PM tomorrow night. Forinstance, communication action module 12 may obtain contextualinformation from a calendar application of the user that indicates ameeting item from 5 PM to 7 PM on Apr. 18, 2014. Communication actionmodule 12 may also obtain contextual information from a clock ofcomputing device 2 indicating that the current date is Apr. 17, 2014.Based on this information, communication action module 12 may determinethat the user has a scheduled meeting until 7 PM tomorrow night. Basedon the information included in the email, metadata or other informationabout the email, and/or the obtained contextual information,communication action module 12 may determine one or more candidateresponses and send the candidate responses to application module 10C.

Application module 10C may receive the candidate responses and sendinformation to UI module 6 to cause UI device 4 to output GUI 160 fordisplay. As shown in the example of FIG. 5, GUI 160 includes responseoptions 162A, 162B, and 162C (collectively, “response options 162”).Each of responses options 162 may correspond to a respective candidateresponse. In the example of FIG. 5, the candidate responses determinedby communication action module 12 may include the candidate response“That sounds great,” “I already have plans,” and “How about 8 p.m.?”

In the example of FIG. 5, computing device 2 may receive an indicationof user input (e.g., selection 164) that selects response option 152A(e.g., corresponding to the candidate response, “that sounds great”).That is, UI device 4 may receive selection 164 and send datarepresenting the input to UI module 6. UI module 6 may receive the dataand provide an indication of the input to application module 10C.Application module 10C may determine that the input corresponds to aselection of response option 162A.

Responsive to receiving the indication of user input that selectsresponse option 162A, application module 10C may cause computing device2 to send the candidate response “That sounds great” as an email. Thatis, responsive to receiving the indication of user input that selects acandidate response, computing device 2 may send the candidate response.Application module 10C may also send an indication of the candidateresponse to communication action module 12.

Communication action module 12 may, in the example of FIG. 5, receivethe indication of the candidate response and determine one or moreoperations to create, modify, or remove a scheduling item related to thesubject or subjects included in the email and/or the selected candidateresponse. For instance, communication action module 12 may parse thelanguage of the selected candidate response using natural languageprocessing techniques, in a fashion similar to the parsing of thereceived communication. Consequently, communication action module 12 maydetermine that the user responded in the affirmative to the socialreservation request. Communication action module 12 may use theindication of affirmation, the indication of a location (e.g., YotteruSushi Restaurant), the indication of location type (e.g., restaurant)and the indication of time (e.g., tomorrow at 9 PM) to determine anoperation. For instance, communication action module 12 may determinethat affirming a location of the restaurant type at a specific time,together, corresponds to a contextually related operation to create areservation with the indicated restaurant at the indicated time.

In the example of FIG. 5, communication action module 12 may execute theoperation. For instance, communication action module 12 may call an APIof a restaurant application or reservation application of computingdevice 2, the call providing the relevant restaurant, restaurantlocation, and time (e.g., Yotteru Sushi Restaurant, 250 E 6th street,and Friday April 18 at 7 PM). The receiving application may execute andsend data to UI module 6 that causes UI device 4 to output GUI 170.

As shown in the example of FIG. 5, GUI 170 may indicate to the user ofcomputing device 2 that computing device 2 is performing or about toperform the operation to create a reservation. GUI 170 includes userselectable elements that may enable the user to modify or cancel theoperation, such as elements 172A and 172B. In some examples, ifcomputing device 2 does not receive an indication of input that selectsone of elements 172A or 172B within a threshold amount of time,computing device 2 (e.g., application module 10C) may perform theoperation and create the reservation.

While described in the example of FIG. 5 as creating a table reservationat a restaurant indicated in the received communication and/or selectedresponse, contextually related operations may create, modify, or removevarious other types of scheduling events related to communicationsubjects. For instance, if communication action module 12 determinesthat the received communication indicates someone will be late to ashared meeting, and the response communication is an acceptance of thisinformation, communication action module 12 may determine an operationto modify the meeting event to start at a later time. As anotherexample, if communication action module 12 determines that the receivedcommunication indicates that a user is invited to a meeting in a firstcity, that the response communication accepts the meeting, and that theuser has a hotel reservation for the same day in a second city,communication action module 12 may determine an operation to cancel thehotel reservation (e.g., in some examples, after prompting the user forconfirmation).

FIG. 6 is a flow diagram illustrating example operations of a computingdevice for performing contextually related operations responsive tosending a response to a received communication, in accordance with oneor more techniques of the present disclosure. For purposes ofillustration only, the example operations of FIG. 6 are described belowwithin the context of FIGS. 1 and 2.

In the example of FIG. 6, computing device 2 may receive a communication(200). Computing device 2 may determine one or more candidate responsesto the received communication, based at least in part on thecommunication (202). For instance, computing device 2 may selectcandidate responses from a number of stored candidate responses, usepredictive techniques to generate candidate responses, and/or send arequest for candidate responses to one or more other computing devices.

Computing device 2 may, in the example of FIG. 6, receive an indicationof user input that selects a candidate response from the one or morecandidate responses (204). Responsive to receiving the indication ofuser input that selects the candidate response, computing device 2 maysend the candidate response (206). For instance, computing device 2 maysend the candidate response to the computing device from which thecommunication was received.

In the example of FIG. 6, computing device 2 may determine an operationthat is contextually related to the candidate response (208). Theoperation may be contextually related to the candidate response based atleast in part on at least one of the candidate response and the receivedcommunication. For instance, computing device 2 may select one or morecontextually related operations from a number of stored operations, usepredictive techniques to determine one or more contextually relatedoperations, or send a request for one or more contextually relatedoperations to one or more other computing devices. Computing device 2may execute the operation (210).

The example operations of FIG. 6 may also be described by one or more ofthe examples below.

A computing device or system configured in accordance with thetechniques described herein may parse out a question from a receivedcommunication, such as an email. The computing device may determinecandidate responses by sending the question through a voice searchmodule. In some examples, the question may be sent with the same accessrights as a user or owner of the system. The voice search module mayprovide a number of possible responses (e.g., ordered by confidence).The system may present the possible responses to a user (e.g., at adisplay device). The system may allow the user to select a response froma list of responses, speak a custom response, or both. In some examples,the system may present the user with an action related to the response.For instance, selecting “thanks for the address, I'm on my way” couldcause the system to open an action to start directions to the receivedaddress.

In some examples, such as where a user of the system grants full trustto a specific contact, the system may automatically reply to messagesreceived from that person without any interaction from the user. Thatis, the system may send responses automatically, in a true “zerointeraction, zero attention” experience. In some examples, the systemmay be enabled to receive candidate responses from other sources, suchas other application modules. For instance, the system may receiveanswers to questions about restaurant bookings from a reservationapplication. In some examples, the user's context can be taken intoconsideration, and response options may be provided that are based onthe context, such as “I'll get back to you after running” or “I'll getback to you after being stuck in traffic.” The techniques describedherein may be applicable to all types of communications, including SMSor text messages, emails, chat messages or IM messages, etc.

In some examples, existing voice search technology and existing actionsmay ease implementation of the techniques described herein. That is, byconnecting the data flows in new ways, a computing device, such as amobile phone may move closer to becoming a true assistant that canhandle information exchange in a fast, platform-independent andefficient way. Because the user is in full control of what is shared,the user's privacy does not need to be compromised. Using techniques ofthe present disclosure, a user may be able to cause a computing deviceto respond to a received communication with complex information in avery short time (e.g., in as few as one or two interactions and in aslittle as one or two seconds). By automatically researching calendarinformation, email information, contact information, and otherinformation, a computing device implementing the techniques describedherein may provide complete information without the user searchingthrough each type of information to determine such a completeunderstanding. Furthermore, by automatically performing actions based onresponses, a computing device implementing the techniques describedherein may allow users to easily perform common tasks.

One example of providing candidate responses may include receiving themessage “Hey, are you doing anything tonight?” and providing theresponses “Meeting with Alex from 6 to 8,” “I'm busy until 8 pm,” and“I'm busy, sorry.” Another example may include receiving the message“Can you meet us for dinner at 9 pm?” and providing the responses “Yes,see you then” and “How about tomorrow night at 8 pm?” Furthermore,responsive to receiving input from a user to select the response “Yes,see you then,” a computing device implementing the techniques describedherein may create a calendar event for dinner at 9 pm. Another exampleof providing candidate responses may include receiving the message “Doyou have Sarah's phone number?” and providing the responses “Sara Louis555-123-4567,” “Sara Lane 555-123-4556,” and “No, sorry.”

Another example of providing candidate responses may include receivingthe message “Please remember to pick up milk” and providing theresponses “Will do,” “Where?,” and “When?” Furthermore, responsive toreceiving input from a user to select the response “Will do,” acomputing device implementing the techniques described herein may add areminder to pick up milk in one hour. Another example of providingcandidate responses may include receiving the message “Where are you?”and providing the responses “At home,” “North Beach,” and “5366Broadway, San Francisco.” Furthermore, responsive to receiving inputfrom a user to select a response, a computing device implementing thetechniques described herein may attach a map to the message. Anotherexample of providing candidate responses may include receiving themessage “How tall is Barack Obama?” and providing the response “6′ 1″(1.85 m).” Another example may include receiving the message “Whichflight are you on?” and providing the response “UA555 SFO-ORD, Delayed 1hour, 10:49 pm, Terminal 3.”

Another example of providing candidate responses may include receivingthe message “Did you call Larry?” and providing the responses “Yes” and“No.” Furthermore, responsive to receiving input from a user to select aresponse, a computing device implementing the techniques describedherein may initiate a phone call to Larry. Another example of providingcandidate responses may include receiving the message “Call me” andproviding the responses “OK” and “I'll call you tonight at 6 pm.”Furthermore, responsive to receiving input from a user to select theresponse “OK,” a computing device implementing the techniques describedherein may initiate a phone call to the sender of the received message.Another example of providing candidate responses may include receivingthe message “When is Matt's party?” and providing the responses“Saturday, 9 pm” and “I don't know.”

EXAMPLE 1

A method comprising: receiving, by a computing device, a communication;determining, based at least in part on the communication, one or morecandidate responses to the communication; receiving, by the computingdevice, an indication of user input that selects a candidate responsefrom the one or more candidate responses; and responsive to receivingthe indication of user input that selects the candidate response:sending, by the computing device, the candidate response, determining,based at least in part on at least one of the candidate response and thecommunication, an operation that is contextually related to thecandidate response, and executing, by the computing device, theoperation.

EXAMPLE 2

The method of example 1, wherein the candidate response has a semanticmeaning, and determining the operation that is contextually related tothe candidate response further comprises: determining one or moreoperations that were previously performed in association with a previousresponse having a semantic meaning that is substantially similar to thesemantic meaning of the candidate response.

EXAMPLE 3

The method of example 2, wherein each respective operation of the one ormore operations is associated with a respective context and whereindetermining the operation that is contextually related to the candidateresponse further comprises: determining a current context of thecomputing device, the current context including a first group of one ormore items of contextual information; and determining for eachrespective operation of the one or more operations, a respective scorebased on a respective second group of items of contextual informationincluded in the respective context associated with the respectiveoperation, the respective second group of items being substantiallysimilar to the first group of one or more items, wherein the operationthat is contextually related to the candidate response is determinedbased at least in part on the respective scores for each of the one ormore operations.

EXAMPLE 4

The method of example 1, wherein determining the operation that iscontextually related to the candidate response further comprises:determining a first group of one or more semantic elements of at leastone of the candidate response and the communication; determining, basedat least in part on the first group of one or more semantic elements, aplurality of operations; and determining a plurality of respectivescores, each respective score from the plurality of respective scoresassociated with a respective operation from the plurality of operations,wherein the respective score associated with the respective operationrepresents a respective degree of similarity, within a range of degreesof similarity, between the first group of one or more semantic elementsand a respective second group of one or more semantic elementsassociated with the respective operation, wherein executing theoperation comprises executing a particular operation, from the pluralityof operations, that is associated with a highest score from theplurality of respective scores.

EXAMPLE 5

The method of example 1, wherein determining the operation that iscontextually related to the candidate response further comprises:determining a semantic meaning of at least one of the candidate responseand the communication; and determining, based at least in part on thesemantic meaning, a registered application, wherein the operation thatis contextually related to the candidate response comprises executingthe registered application based on information included in at least oneof the candidate response and the communication.

EXAMPLE 6

The example of any of examples 1-5, wherein determining the one or morecandidate responses to the communication further comprises: determininga semantic meaning of the communication; and determining one or moreresponses that were previously selected to respond to a previouslyreceived communication having a semantic meaning that is substantiallysimilar to the semantic meaning of the communication.

EXAMPLE 7

The method of example 6, wherein each respective response of the one ormore responses is associated with a respective context and whereindetermining the one or more candidate responses to the communicationfurther comprises: determining a current context of the computingdevice, the current context including a first group of one or more itemsof contextual information; and determining for each respective responseof the one or more responses, a respective score based on a second groupof items of contextual information included in the respective contextassociated with the respective response, the respective second group ofitems being substantially similar to the first group of one or moreitems, wherein the one or more candidate responses to the communicationare determined based at least in part on the respective scores for eachof the one or more responses.

EXAMPLE 8

The method of any of examples 1-7, further comprising: determining acurrent context of the computing device, the current context indicatingat least one of: a location of the computing device, a time determinedby the computing device, one or more applications installed at thecomputing device, one or more applications currently executing at thecomputing device, one or more networks available to the computingdevice, one or more other computing devices in proximity to thecomputing device, an operating mode of the computing device, an ambienttemperature around the computing device, an ambient noise level aroundthe computing device, an ambient light level around the computingdevice, an acceleration of the computing device, a name of a user of thecomputing device, a user identifier of the user, a social media networkservice account associated with the user, a calendar of the user, andone or more social contacts of the user, wherein determining the one ormore candidate responses to the communication is based at least in parton the current context of the computing device.

EXAMPLE 9

The method of any of examples 1-8, wherein at least one of the candidateresponse and the communication indicates a location, and whereindetermining the operation comprises: determining travel directions tothe location; and outputting at least a portion of the traveldirections.

EXAMPLE 10

The method of any of examples 1-8, wherein at least one of the candidateresponse and the communication indicates a location, and whereindetermining the operation comprises: determining weather informationabout the location; and outputting at least a portion of the weatherinformation.

EXAMPLE 11

The method of examples 1-8, wherein determining the operation that iscontextually related to the candidate response comprises at least oneof: creating a scheduling item managed by a scheduling application;modifying the scheduling item managed by the scheduling application; andremoving the scheduling item managed by the scheduling application.

EXAMPLE 12

The method of example 11, wherein the scheduling item is a calendarevent, and wherein creating, modifying, or removing the scheduling itemcomprises: creating the calendar event managed by a calendarapplication; modifying the calendar event managed by the calendarapplication; or removing the calendar event managed by the calendarapplication.

EXAMPLE 13

The method of example 11, wherein the scheduling item is a reservation,and wherein creating, modifying, or removing the scheduling itemcomprises: creating the reservation managed by a reservationapplication; modifying the reservation managed by the reservationapplication; or removing the reservation managed by the reservationapplication.

EXAMPLE 14

The method of example 13, wherein the reservation comprises at least oneof: a table reservation; a room reservation; a hotel reservation; aflight reservation; and a rental car reservation.

EXAMPLE 15

The method of any of examples 1-14, further comprising: determining atleast one application module installed at the computing device; anddetermining the operation that is contextually related to the candidateresponse based at least in part on the at least one application modulebeing installed at the computing device.

EXAMPLE 16

The example of any of claims 1-15, further comprising determining acurrent context of the computing device, the current context comprisingone or more items of contextual information; determining a time framethroughout which a context of at least one of the computing device andanother computing device comprised at least one item of contextualinformation included in the current context; and determining a previousoperation that was performed by the at least one of the computing deviceand another computing device during the time frame.

EXAMPLE 17

The method of any of examples 1-16, wherein determining the operationthat is contextually related to the candidate response comprises:determining at least one of: information that is included in the atleast one of the candidate response and the communication, metadataabout the at least one of the candidate response and the communication,one or more other communications, and contextual information associatedwith the at least one of the candidate response and the communication;and determining the operation that is contextually related to thecandidate response based at least in part on at least one of: theinformation that is included in the at least one of the candidateresponse and the communication, the metadata about the at least one ofthe candidate response and the communication, the one or more othercommunications, and the contextual information associated with the atleast one of the candidate response and the communication.

In one or more examples, the functions described may be implemented inhardware, software, firmware, or any combination thereof. If implementedin software, the functions may be stored on or transmitted over, as oneor more instructions or code, a computer-readable medium and executed bya hardware-based processing unit. Computer-readable media may includecomputer-readable storage media, which corresponds to a tangible mediumsuch as data storage media, or communication media, which includes anymedium that facilitates transfer of a computer program from one place toanother, e.g., according to a communication protocol. In this manner,computer-readable media generally may correspond to (1) tangiblecomputer-readable storage media, which is non-transitory or (2) acommunication medium such as a signal or carrier wave. Data storagemedia may be any available media that can be accessed by one or morecomputers or one or more processors to retrieve instructions, codeand/or data structures for implementation of the techniques described inthis disclosure. A computer program product may include acomputer-readable storage medium.

By way of example, and not limitation, such computer-readable storagemedia can comprise RAM, ROM, EEPROM, CD-ROM or other optical diskstorage, magnetic disk storage, or other magnetic storage devices, flashmemory, or any other medium that can be used to store desired programcode in the form of instructions or data structures and that can beaccessed by a computer. Also, any connection is properly termed acomputer-readable medium. For example, if instructions are transmittedfrom a website, server, or other remote source using a coaxial cable,fiber optic cable, twisted pair, digital subscriber line (DSL), orwireless technologies such as infrared, radio, and microwave, then thecoaxial cable, fiber optic cable, twisted pair, DSL, or wirelesstechnologies such as infrared, radio, and microwave are included in thedefinition of medium. It should be understood, however, thatcomputer-readable storage media and data storage media do not includeconnections, carrier waves, signals, or other transient media, but areinstead directed to non-transient, tangible storage media. Disk anddisc, as used herein, includes compact disc (CD), laser disc, opticaldisc, digital versatile disc (DVD), floppy disk and Blu-ray disc, wheredisks usually reproduce data magnetically, while discs reproduce dataoptically with lasers. Combinations of the above should also be includedwithin the scope of computer-readable media.

Instructions may be executed by one or more processors, such as one ormore digital signal processors (DSPs), general purpose microprocessors,application specific integrated circuits (ASICs), field programmablelogic arrays (FPGAs), or other equivalent integrated or discrete logiccircuitry. Accordingly, the term “processor,” as used herein may referto any of the foregoing structure or any other structure suitable forimplementation of the techniques described herein. In addition, in someaspects, the functionality described herein may be provided withindedicated hardware and/or software modules. Also, the techniques couldbe fully implemented in one or more circuits or logic elements.

The techniques of this disclosure may be implemented in a wide varietyof devices or apparatuses, including a wireless handset, an integratedcircuit (IC) or a set of ICs (e.g., a chip set). Various components,modules, or units are described in this disclosure to emphasizefunctional aspects of devices configured to perform the disclosedtechniques, but do not necessarily require realization by differenthardware units. Rather, as described above, various units may becombined in a hardware unit or provided by a collection ofinteroperative hardware units, including one or more processors asdescribed above, in conjunction with suitable software and/or firmware.

Various examples have been described. These and other examples arewithin the scope of the following claims.

1-20. (canceled)
 21. A method comprising: receiving, by a computingdevice, a communication; determining, based at least in part on thecommunication, one or more candidate responses to the communication;receiving, by the computing device, an indication of user input thatselects a candidate response from the one or more candidate responses;and responsive to receiving the indication of user input that selectsthe candidate response: sending, by the computing device, the candidateresponse; determining, based at least in part on a group of semanticelements associated with at least one of the candidate response or thecommunication, one or more respective scores for one or moreapplications; selecting, from the one or more applications, anapplication corresponding to a highest score from the one or morerespective scores; and executing, based at least in part on at least oneof the candidate response or the communication, an operation with theapplication.
 22. The method of claim 21, wherein the candidate responsehas a semantic meaning, the method comprising: determining the operationwas previously performed in association with a previous response havinga semantic meaning that is substantially similar to the semantic meaningof the candidate response.
 23. The method of claim 22, furthercomprising: determining a current context of the computing device, thecurrent context including a first group of one or more items ofcontextual information; and determining, a score based on a second groupof items of contextual information included in a context associated withthe operation, the second group of items being substantially similar tothe first group of one or more items, wherein the operation that iscontextually related to the candidate response is executed by thecomputing device based at least in part on the score for the operation.24. The method of claim 21, wherein the group of semantic elements is afirst group of semantic elements, the method further comprising:determining the first group of one or more semantic elements from atleast one of the candidate response and the communication; determining,based at least in part on the first group of one or more semanticelements, a plurality of operations that includes the operation; anddetermining a plurality of respective scores, each respective score fromthe plurality of respective scores associated with a respectiveoperation from the plurality of operations, wherein the respective scoreassociated with the respective operation represents a respective degreeof similarity, within a range of degrees of similarity, between thefirst group of one or more semantic elements and a respective secondgroup of one or more semantic elements associated with the respectiveoperation, wherein the operation, executed with the application, isassociated with a highest score from the plurality of respective scores.25. The method of claim 21, further comprising: determining a semanticmeaning of at least one of the candidate response and the communication;and determining, based at least in part on the semantic meaning, theapplication, wherein executing the operation comprises executing theoperation application based on information included in at least one ofthe candidate response and the communication.
 26. The method of claim21, wherein determining one or more candidate responses to thecommunication comprises: determining a semantic meaning of thecommunication; and determining one or more responses that werepreviously selected to respond to a previously received communicationhaving a semantic meaning that is substantially similar to the semanticmeaning of the communication.
 27. The method of claim 21, wherein eachrespective response of the one or more responses is associated with arespective context, and wherein determining the one or more candidateresponses to the communication further comprises: determining a currentcontext of the computing device, the current context including a firstgroup of one or more items of contextual information; and determiningfor each respective response of the one or more responses, a respectivescore based on a respective second group of items of contextualinformation included in the respective context associated with therespective response, the respective second group of items beingsubstantially similar to the first group of one or more items, whereinthe one or more candidate responses to the communication are determinedbased at least in part on the respective scores for each of the one ormore responses.
 28. The method of claim 21, further comprising:determining a current context of the computing device, the currentcontext indicating at least one of: a location of the computing device,a time determined by the computing device, one or more applicationsinstalled at the computing device, one or more applications currentlyexecuting at the computing device, one or more networks available to thecomputing device, one or more other computing devices in proximity tothe computing device, an operating mode of the computing device, anambient temperature around the computing device, an ambient noise levelaround the computing device, an ambient light level around the computingdevice, an acceleration of the computing device, a name of a user of thecomputing device, a user identifier of the user, a social media networkservice account associated with the user, a calendar of the user, andone or more social contacts of the user, wherein determining the one ormore candidate responses to the communication is based at least in parton the current context of the computing device.
 29. The method of claim21, wherein at least one of the candidate response and the communicationindicates a location, the method further comprising: executing theoperation comprises at least one of: determining travel directions tothe location; and outputting at least a portion of the traveldirections.
 30. The method of claim 21, wherein executing the operationcomprises at least one of: creating a scheduling item managed by ascheduling application; modifying the scheduling item managed by thescheduling application; and removing the scheduling item managed by thescheduling application.
 31. The method of claim 21, wherein executingthe operation further comprises: determining at least one of:information that is included in the at least one of the candidateresponse and the communication, metadata about the at least one of thecandidate response and the communication, one or more othercommunications, and contextual information associated with the at leastone of the candidate response and the communication; and executing theoperation based at least in part on at least one of: the informationthat is included in the at least one of the candidate response and thecommunication, the metadata about the at least one of the candidateresponse and the communication, the one or more other communications,and the contextual information associated with the at least one of thecandidate response and the communication.
 32. A computing devicecomprising: one or more computer processors; and a memory comprisinginstructions that when executed by the one or more computer processorscause the one or more computer processors to: receive a communication;determine, based at least in part on the communication, one or morecandidate responses to the communication; receive an indication of userinput that selects a candidate response from the one or more candidateresponses; and responsive to receiving the indication of user input thatselects the candidate response: send the candidate response; determine,based at least in part on a group of semantic elements associated withat least one of the candidate response or the communication, one or morerespective scores for one or more applications; select, from the one ormore applications, an application corresponding to a highest score fromthe one or more respective scores; and execute, based at least in parton at least one of the candidate response or the communication, anoperation with the application.
 33. The computing device of claim 32,wherein the candidate response has a semantic meaning, wherein thememory comprises instructions that when executed by the one or morecomputer processors cause the one or more computer processors to:determine the operation was previously performed in association with aprevious response having a semantic meaning that is substantiallysimilar to the semantic meaning of the candidate response.
 34. Thecomputing device of claim 32, wherein the memory comprises instructionsthat when executed by the one or more computer processors cause the oneor more computer processors to: determining a current context of thecomputing device, the current context including a first group of one ormore items of contextual information; and determining, a score based ona second group of items of contextual information included in a contextassociated with the operation, the second group of items beingsubstantially similar to the first group of one or more items, whereinthe operation that is contextually related to the candidate response isexecuted by the computing device based at least in part on the score forthe operation.
 35. The computing device of claim 32, wherein the groupof semantic elements is a first group of semantic elements, wherein thememory comprises instructions that when executed by the one or morecomputer processors cause the one or more computer processors to:determine the first group of one or more semantic elements from at leastone of the candidate response and the communication; determine, based atleast in part on the first group of one or more semantic elements, aplurality of operations that includes the operation; and determine aplurality of respective scores, each respective score from the pluralityof respective scores associated with a respective operation from theplurality of operations, wherein the respective score associated withthe respective operation represents a respective degree of similarity,within a range of degrees of similarity, between the first group of oneor more semantic elements and a respective second group of one or moresemantic elements associated with the respective operation, wherein theoperation, executed with the application, is associated with a highestscore from the plurality of respective scores.
 36. The computing deviceof claim 32, wherein the memory comprises instructions that whenexecuted by the one or more computer processors cause the one or morecomputer processors to: determine a semantic meaning of at least one ofthe candidate response and the communication; and determine, based atleast in part on the semantic meaning, the application, whereinexecuting the operation comprises executing the operation applicationbased on information included in at least one of the candidate responseand the communication.
 37. The computing device of claim 36, whereineach respective response of the one or more responses is associated witha respective context, wherein the memory comprises instructions thatwhen executed by the one or more computer processors cause the one ormore computer processors to: determine a current context of thecomputing device, the current context including a first group of one ormore items of contextual information; and determine, for each respectiveresponse of the one or more responses, a respective score based on arespective second group of items of contextual information included inthe respective context associated with the respective response, therespective second group of items being substantially similar to thefirst group of one or more items, wherein the one or more candidateresponses to the communication are determined based at least in part onthe respective scores for each of the one or more responses.
 38. Thecomputing device of claim 32, wherein the group of semantic elements isa first group of semantic elements, wherein the memory comprisesinstructions that when executed by the one or more computer processorscause the one or more computer processors to: determine the first groupof one or more semantic elements from at least one of the candidateresponse and the communication; determine, based at least in part on thefirst group of one or more semantic elements, a plurality of operationsthat includes the operation; and determine a plurality of respectivescores, each respective score from the plurality of respective scoresassociated with a respective operation from the plurality of operations,wherein the respective score associated with the respective operationrepresents a respective degree of similarity, within a range of degreesof similarity, between the first group of one or more semantic elementsand a respective second group of one or more semantic elementsassociated with the respective operation, wherein the operation,executed with the application, is associated with a highest score fromthe plurality of respective scores.
 39. The computing device of claim32, wherein the memory comprises instructions that when executed by theone or more computer processors cause the one or more computerprocessors to: determine a semantic meaning of at least one of thecandidate response and the communication; and determine, based at leastin part on the semantic meaning, the application, wherein executing theoperation comprises executing the operation application based oninformation included in at least one of the candidate response and thecommunication
 40. A computer-readable storage medium encoded withinstructions that, when executed, cause at least one processor to:receive a communication; determine, based at least in part on thecommunication, one or more candidate responses to the communication;receive an indication of user input that selects a candidate responsefrom the one or more candidate responses; and responsive to receivingthe indication of user input that selects the candidate response: sendthe candidate response; determine, based at least in part on a group ofsemantic elements associated with at least one of the candidate responseor the communication, one or more respective scores for one or moreapplications; select, from the one or more applications, an applicationcorresponding to a highest score from the one or more respective scores;and execute, based at least in part on at least one of the candidateresponse or the communication, an operation with the application.